ADR-0011: Version Numbers

Status:	ACCEPTED
Date:	2022-03-08
Author(s):	Max Maass max.maass@iteratec.com

Context

Software version numbers should serve as an indicator of compatibility between different versions, both during operation and while updating to a newer release. At the moment, the secureCodeBox is following the Semantic Versioning (semver) approach. According to semver, version numbers are of the format MAJOR.MINOR.PATCH, and the different places are incremented as follows:

1. MAJOR version when you make incompatible API changes,
2. MINOR version when you add functionality in a backwards compatible manner, and
3. PATCH version when you make backwards compatible bug fixes.

However, the architecture of secureCodeBox, with its operator and many individual scanners that can be installed and used separately, makes this seemingly simple versioning system more difficult. For example, we need to answer the following questions:

1. Are breaking changes in the parameterization of a scanner (e.g. nmap, Gitleaks) a breaking change for the entire secureCodeBox project?
2. Are changes to the output format of a single scanner a breaking change for the entire project?
3. In an environment where operator and scanners aren't necessarily using the same version number because the operator is controlled by a different team than the scanners (a setup that we want to support), how do we indicate compatibility between different versions of operator and scanner?

Depending on how these questions are answered, different versioning schemes are possible.

Option 1: SemVer With Major Version Indicating Overall Compatibility Of All Components

The basic premise for this versioning scheme would be:

Any change that requires manual actions from at least one user of the SCB to keep existing workflows running after an update is considered a breaking change and requires a MAJOR release.

This manual action may include making changes to scan definitions, or to systems that are ingesting data from the SCB findings of a scanner. It sees the entire secureCodeBox as one large piece of software with many components that are all equally important to the overall compatibility, and where all components are (usually) updated in lockstep. As illustration, here are a few examples and what kind of release they would require:

Action	Version
A scanner changes how it is parameterized	Major
A scanner removes data from its output or makes other changes affecting the findings	Major
The SCB makes changes to the findings format of one scanner (e.g., renaming a key)	Major
The SCB makes breaking changes to the CRDs (renaming or removing fields)	Major
The SCB makes backwards-compatible changes to the CRDs (adding new fields)	Minor
The SCB fixes a small bug in the operator or a scanner	Patch

Advantages

1. MAJOR versions indicate that manual action may be required to keep existing workflows running and that the users should read the changelog. However, it may turn out that the breaking change does not apply to the users' environment (e.g., because they are not using a specific scanner), in which case no manual action may be required.
2. MINOR versions can be installed with the expectation that no manual action will be required to keep existing workflows working. However, manual action may be required to benefit from new features (due to changes to the CRDs that need to be manually installed).
3. Frequent MAJOR releases lower the inhibition to make larger changes. At the moment, many proposed larger changes are pushed back to "the next major version", without actively planning towards releasing such a version. Making major releases more common makes it easier to include smaller breaking changes that improve the consistency of the system (e.g., in the case of the output format).
4. It is compliant with the expectations of users that expect SCB to be versioned like a single monolithic piece of software.

Disadvantages

1. The MAJOR version no longer indicates compatibility between operator and scanner versions. In environments where operator and scanners are updated separately by different groups, this increases complexity greatly (we know that such environments exist, and want to support them). It also raises the question how this compatibility will be documented instead.
2. The versioning scheme does not distinguish between changes that are breaking to a small subset of users, and breaking to all users. This makes it harder for users to distinguish based on the version number alone if an update is going to take 5 minutes (because all the breakage is in a component that they do not use) or 5 hours (because there are large changes to the CRDs, like there were between SCBv2 and SCBv3).

Option 2: SemVer With Major Version Indicating Operator Compatibility

The basic premise for this versioning scheme would be:

Any change that breaks compatibility between operator and scanner is considered a breaking change requiring a MAJOR release. Breaking changes to individual scanners are considered non-breaking for the overall system and instead use a MINOR release.

This approach sees the secureCodeBox as a platform with independent components, more akin to an operating system or kernel than a monolithic piece of software. The MAJOR version number indicates compatibility between the operator and scanners, while MINOR version changes can still be breaking to some users (in which case this will be denoted prominently at the top of the release notes). As illustration, here are a few examples and what kind of release they would require:

Action	Version
A scanner changes how it is parameterized	Minor
A scanner removes data from its output or makes other changes affecting the findings	Minor
The SCB makes changes to the findings format of one scanner (e.g., renaming a key)	Minor
The SCB makes breaking changes to the CRDs (renaming or removing fields)	Major
The SCB makes backwards-compatible changes to the CRDs (adding new fields)	Minor
The SCB fixes a small bug in the operator or a scanner	Patch

Advantages

1. MAJOR versions indicate that a joint upgrade of operator and scanners is required, which is highly relevant in environments where operator and scanners are maintained by different teams (we know that these environments exist, and want to support them). This obviates the need for a detailed compatibility matrix between scanner and operator versions.
2. It is compliant to the expectations of users that expect SCB to be versioned like a platform or operating system.

Disadvantages

1. MINOR version changes can be breaking to some users, forcing everyone to read the changelogs of all intermediate minor version changes when upgrading a scanner.

Option 3: Version Number With Architecture Prefix For Operator Compatibility

A different variant of the previous option would be to prefix the version number with an ARCHITECTURE (ARCH) number, so the final versioning would be ARCH.MAJOR.MINOR.PATCH. In that case, the table would look like this:

Action	Version
A scanner changes how it is parameterized	Major
A scanner removes data from its output or makes other changes affecting the findings	Major
The SCB makes changes to the findings format of one scanner (e.g., renaming a key)	Major
The SCB makes breaking changes to the CRDs (renaming or removing fields)	Arch
The SCB makes backwards-compatible changes to the CRDs (adding new fields)	Minor
The SCB fixes a small bug in the operator or a scanner	Patch

This could be considered "playing tricks" (by just adding a digit and relabeling the meaning of the positions) and would deviate from semver, but would allow us to use the ARCH number to denote operator compatibility, while the MAJOR version denotes scanner compatibility, the MINOR indicates feature additions, and the PATCH indicates bugfixes.

Advantages

1. ARCH versions indicate that a joint upgrade of operator and scanners is required, which is highly relevant in environments where operator and scanners are maintained by different teams. This obviates the need for a detailed compatibility matrix between scanner and operator versions.
2. MAJOR versions indicate that there have been breaking changes in a component (a scanner, a hook, ...), but compability with other components remains. The exact breakage and how to address it is communicated prominently in the changelog.
3. MINOR and PATCH versions can be installed without worrying about compatibility for existing scans.

Disadvantages

1. The proposal deviates from the SemVer standard and is thus unexpected for people who do not know about it.
2. Helm will likely not accept version numbers in this format, as it is very strict about adherence to semver. Thus, this solution cannot be implemented while using Helm.

Option 4: Semantic Versioning, Separate Versioning For Components

For completeness sake, we also include this option, in which each component is versioned separately, so that a breaking change in one component only changes the version number of that component. We previous discarded this idea because it would require complex documentation of which scanner versions work with which operator versions. We will thus not consider this proposal in greater detail here.

Option 5: SemVer With Major Version Indicating Operator Compatibility + Included Version of "Embedded Dependency"

(Based on Suggestion by @EndPositive) This option expands on option 2, but adds an additional version of embedded dependencies (primarily of scanner) in the build metadata section of the semver version.

Example Version Numbers:

• operator: 3.42.0
• lurker: 3.42.0
• nmap: 3.42.0+7.92-r2
• amass: 3.42.0+v3.16.0
• zap: 3.42.0+2.11.1

Currently undecided:

• Same structure for hooks?
  • e.g. embedded version / version range of supported DefectDojo version in build meta
  • might not be necessary / possible for every hook. And thus make the hook versioning schema inconsistent

Action	Version
A scanner changes how it is parameterized	Major
A scanner removes data from its output or makes other changes affecting the findings	Major
The SCB makes changes to the findings format of one scanner (e.g., renaming a key)	Major
The SCB makes breaking changes to the CRDs (renaming or removing fields)	Arch
The SCB makes backwards-compatible changes to the CRDs (adding new fields)	Minor
The SCB fixes a small bug in the operator or a scanner	Patch
A scanner changes its own version	Updated Build Meta

This makes the proposal still not completely Semver compliant as the command line flags of the scanner can still break in minor releases, but these can be easily detected as the user will notice that the version number of the scanner has jumped a major release.

Advantages

1. MAJOR versions indicate that a joint upgrade of operator and scanners is required, which is highly relevant in environments where operator and scanners are maintained by different teams (we know that these environments exist, and want to support them). This obviates the need for a detailed compatibility matrix between scanner and operator versions.
2. It is compliant to the expectations of users that expect SCB to be versioned like a platform or operating system.
3. No additional version number scheme has to be maintained Tool version can be automatically be taken from the helm charts app version and appended to the chart version during packaging.

Disadvantages

1. MINOR version changes can be breaking to some users,, but can be identified by looking at the embedded version number
2. Version numbers can look confusing and require an explanation
3. Upgrading scanners in GitOps systems like Flux or ArgoCD is no longer possible using search-and-replace, as it requires knowing the version number of the scanner in addition to that of the secureCodeBox.

Decision

There is no perfect solution for this - either we have a large number of "breaking" changes that are only breaking for some subset of users, or we have minor releases that are breaking to some users. Under these circumstances, we prefer a solution that allows us to easily indicate compatibility between operator and scanner, and makes it easy to use in a GitOps environment. We thus choose to use option 2: Breaking changes for individual scanners can be part of a minor release, and only globally relevant changes require a major release. This is in line with how we have practiced versioning over the last months, and simply makes it more explicit.

Consequences

In the future, minor releases can contain changes that are breaking to some users. These changes will be highlighted in the release notes. Major releases will be used to indicate larger compatibility-breaking changes.