Skip to content

Release Process

Target audience: Maintainers Goal: Specification of how StageX releases are created, validated, signed, and published.

Overview

StageX follows a release process designed to eliminate single points of failure at every stage. No single maintainer, build machine, or signing key can unilaterally produce or publish a release. Every artifact must be independently reproduced on hardware from at least two different CPU vendors, individually signed by two or more maintainers, and merged through a structured branch flow with signed merge commits.

This process is enforced through a combination of automation (make targets and shell scripts), cryptographic controls (PGP signing, sigstore-format payloads), and social policy (mandatory review quorum, PR adoption protocol). The result is a cryptographically auditable chain from source commit to published OCI image.

Release Versioning

Releases follow a YYYY.MM.N version scheme:

Component Meaning Example
YYYY Four-digit year 2026
MM Two-digit month 01
N Monotonic release revision within the month 0, 1, 2

The version is auto-generated by src/gen-version.sh, which queries the latest git tag matching the YYYY.MM.N pattern:

  • If the latest tag matches the current year and month, the release revision (N) is incremented by 1.
  • Otherwise, a new release starts at N=0 for the current year and month.

Releases may be set explicitly via the RELEASE environment variable:

make RELEASE=2026.01.0 <target>

Pre-Release Workflow

Before creating a release branch, the release engineer prepares the build environment:

  1. Seed the build cache — make preseed pulls the last published release images into the local OCI store. This accelerates the full tree build by reusing unchanged layers.
  2. Verify system compatibility — make compat checks that all required tooling (bash, Docker/BuildKit, jq, GPG) meets minimum version requirements. See Makefile Targets & Environment for the full version table.
  3. Verify local build against committed digests — make verify reads the committed digest files and validates that locally built OCI manifests match the expected digests.

Hardware requirements for full tree builds are specified in the System Requirements. Cold builds on adequate hardware (~16 cores, 32 GB RAM, 650 GB SSD) take approximately 24 hours.

The release engineer MUST be on the staging branch with a clean working tree (no uncommitted changes) before proceeding.

Creating a Release

Branch Preparation

The release branch is created with the prep-release-branch Make target:

make prep-release-branch

This invokes src/prep-release-branch.sh, which:

  1. Fetches the latest repository state.
  2. If RELEASE is not set, auto-generates it via gen-version.sh.
  3. Validates the current branch is staging.
  4. Creates a new branch release/YYYY.MM.N.

Digest Generation

After branching, the release engineer generates fresh digests:

make digests

src/digests.py scans out/**/index.json for every built package, resolves each OCI index to its manifest digest, and writes per-stage digest files to digests/<stage>.txt. The format is:

<sha256-manifest-digest> <stage>-<package-name>

The new digest files are committed to the release branch and a pull request is opened against the main branch on the Git forge. The PR description MUST summarize all changes since the latest release.

Building and Independent Reproduction

The full package tree must be reproduced on hardware from at least two different CPU vendors (e.g., AMD and Intel). This requirement detects vendor-specific microarchitectural backdoors and ensures that build results are portable and deterministic.

Each maintainer participating in the release independently:

  1. Checks out the release branch.
  2. Runs make preseed to seed the build cache.
  3. Runs make (or make <stage>) to build the entire tree.
  4. Compares their local digests against the committed digests.

If all maintainers produce identical digests, the build is considered reproducible. See Reproduce Builds Locally for detailed reproduction instructions.

Signing

Signature Format

StageX uses the Container Signature Format (sigstore), a standard JSON payload with the following structure:

{
  "critical": {
    "identity": {
      "docker-reference": "<registry>/<package>:<tag>"
    },
    "image": {
      "docker-manifest-digest": "sha256:<manifest-digest>"
    },
    "type": "atomic container signature"
  },
  "optional": {}
}

Each maintainer signs this payload with their PGP key. The resulting binary detached signature is written to a file named signature-<N> under:

signatures/<registry>/<package>@sha256=<manifest-digest>/

See Decentralized Multi-Sig Signing for the full signing protocol.

Signing Process

The release engineer runs:

make sign

src/sign-all.sh performs the following:

  1. Validates the working tree is clean.
  2. Clones or updates the signatures repository.
  3. Creates or checks out a release/YYYY.MM.N branch in the signatures repo.
  4. Iterates over all digests in digests/*.txt and invokes src/sign.sh for each.
  5. Each sign.sh invocation:
  6. Resolves the OCI index to find the manifest digest.
  7. Computes the target directory: signatures/<registry>/<package>@sha256=<manifest-digest>.
  8. Checks whether the calling maintainer has already signed (deduplication by PGP fingerprint).
  9. If not, constructs the sigstore JSON payload and signs it with $GPG_SIGN --sign --no-armor.

Quorum Requirement

A minimum of two independent maintainers MUST contribute their signatures for every digest in a release. This is enforced at publish time by the publish-* targets:

signum="$$(ls -1 signatures/stagex/<package>@sha256=<digest> | wc -l)"
[ $${{signum}} -ge 2 ] || { echo "Error: Minimum signatures not met"; exit 1; }

Each maintainer runs make sign independently after reproducing the build. The signatures are committed to the signatures repository via a pull request. The release engineer merges the signatures PR once quorum is reached.

See Verify Multi-Signature Attestations for the consumer-side verification workflow.

Publishing

Once all packages have at least two signatures, the release engineer publishes:

make RELEASE=2026.01.0 publish

The publish-* targets (generated by src/targets.py) for each package:

  1. Load the OCI layout from out/<stage>-<name>/ into the local Docker Engine.
  2. Tag the image with three tags:
  3. :latest
  4. :sx<RELEASE> (e.g., :sx2026.01.0)
  5. :<version> (e.g., :2026.01.0)
  6. Push each tag to two registries:
  7. docker.io/stagex/<package>
  8. quay.io/stagex/<package>

Push operations include a retry loop (via the push-image macro in src/macros.mk) that retries on failure with a 5-second delay.

Branch Flow

Every release follows a mandatory branch flow that maintains cryptographic auditability:

PR → staging → release/YYYY.MM.N → staging → main

Flow Stages

  1. Pull Request — Every change originates as a PR from a feature branch. Non-maintainer contributions must be "adopted" by a maintainer with an empty signed commit (git commit --allow-empty -m "Adopt").
  2. First staging merge — The PR is merged into staging via a signed merge commit.
  3. Release branch — prep-release-branch creates release/YYYY.MM.N from staging. The release engineer commits digests and runs make sign.
  4. Second staging merge — Once all signatures are collected, the release branch is merged back into staging via a signed merge commit.
  5. Main merge — staging is merged into main via a signed merge commit. The commit message MUST include the release name (e.g., Merge branch 'staging' (Release 2026.01.0)).
  6. Tagging — A signed release tag is created on main with a list of major changes and contributors.

Merge Commit Enforcement

All merges MUST use signed merge commits. Configure local git:

git config merge.ff false
git config commit.gpgsign true

Committing directly to staging or the release branch (bypassing the PR flow) removes the ability to track who approves contributions and when. This is prohibited.

Post-Release Patches

If changes are required after a release branch has been merged to main that cannot wait for the next monthly release:

  1. Cherry-pick the fix to the release branch.
  2. Re-build and re-sign the affected packages.
  3. Collect quorum signatures on the new digests.
  4. Publish the updated images.
  5. Create a new signed tag on the release branch.

Post-release patches MUST NOT be merged to staging or main — the release branch is the only target. This prevents incomplete or partially reviewed changes from entering the main development line before the next full release cycle.

Maintainer Responsibilities

PGP Key Management

Maintainers handling release signing MUST adhere to the following key hygiene requirements:

  • Air-gapped key storage: Private key material MUST never be exposed to an internet-connected environment.
  • Hardware-backed keys: Keys MUST reside on high-quality smart cards (YubiKey 5 series, NitroKey 3) or equivalent isolation (Split GPG on Qubes).
  • PIN protection: Smart card PINs MUST use non-default passwords for both user and admin PINs.
  • Touch/Interaction requirement: All PGP operations MUST require physical touch (smart cards) or explicit user interaction (Split GPG).

See the Maintainer's Handbook for complete key management procedures.

Review and Adoption

  • Every source change MUST be reviewed for malicious code, source URL legitimacy, absence of pre-compiled binaries, bit-for-bit reproducibility, and known vulnerability status.
  • Non-maintainer contributions MUST be explicitly adopted by a maintainer via an empty signed commit (git commit --allow-empty -m "Adopt").
  • Two maintainers MUST sign off on every change. A single maintainer merging is insufficient.
  • All adoptions MUST be the last commit in the PR branch before merge.

Signing Artifacts

Signing release build artifacts constitutes a cryptographic attestation that the signing maintainer has:

  1. Independently reproduced the build.
  2. Confirmed that all package hashes match the other maintainer's results.
  3. Either personally reviewed all new code changes, or has observed signed evidence that at least two maintainers have reviewed all code changes.

Release Checklist Summary

Step Action Command/Artifact
1 Seed build cache make preseed
2 Verify tooling make compat
3 Create release branch make prep-release-branch (from staging)
4 Build full tree make (see hardware requirements)
5 Generate digests make digests
6 Commit digests, open PR Git forge PR targeting main
7 Independent reproduction Other maintainers build + compare digests
8 Sign all digests make sign (each maintainer independently)
9 Merge signatures PR Signatures repository
10 Merge release → staging Signed merge commit
11 Merge staging → main Signed merge commit (with release name)
12 Create signed tag git tag -s with changes/contributors
13 Publish images make RELEASE=<ver> publish
14 Update website Per website repository instructions

See Also