Decentralized Multi-Sig Signing

Introduction

Multi-signature signing is a cryptographic mechanism requiring more than one independent party to attest to the authenticity and integrity of an artifact before it is accepted as valid. In the context of container images, a multi-signature attestation cryptographically links a specific image digest to a specific package name, with the attestation countersigned by multiple independent verifiers.

StageX uses the Container Signature Format, a standard JSON payload defined by the containers/image project. Each maintainer who independently builds a package and confirms its digest matches other maintainers' results signs this payload with their PGP key. The resulting binary detached signatures are stored in a lookaside repository, separate from both the source code repository and the image registry.

This approach ensures that no single maintainer, build machine, or signing key has the authority to publish a StageX artifact unilaterally. A quorum of at least two independent signatures is required before any image can be released.

The Signing Protocol

The foundation of StageX's signing system is a JSON payload conforming to the Container Signature Format. The payload has three required fields within the critical object:

identity.docker-reference -- the registry and package name (e.g., docker.io/stagex/pallet-rust)
image.docker-manifest-digest -- the SHA-256 digest of the image manifest being attested (e.g., sha256:2fbe7b1...)
type -- always "atomic container signature"

An optional field is present but unused in StageX's current implementation.

The sign.sh script constructs this payload and signs it with PGP:

{
  "critical": {
    "identity": {
      "docker-reference": "docker.io/stagex/pallet-rust"
    },
    "image": {
      "docker-manifest-digest": "sha256:2fbe7b164dd92edb9c1096152f6d27592d8a69b1b8eb2fc907b5fadea7d11668"
    },
    "type": "atomic container signature"
  },
  "optional": {}
}

The payload is piped through $GPG_SIGN --sign --no-armor to produce a binary detached PGP signature. The --no-armor flag ensures the output is binary rather than ASCII-armored, as required by the Container Signature Format specification.

Signatures are stored as sequentially numbered files: signature-1, signature-2, and so on. The filename is determined dynamically: if signature-1 exists, the script finds the highest existing number and increments it. This auto-incrementing scheme supports an arbitrary number of signers per artifact without filename conflicts.

The get_primary_fp function in sign.sh resolves a signing key fingerprint to the primary key fingerprint, ensuring that subkey signatures are correctly attributed. The dir_has_no_sig function checks whether the current maintainer has already signed the artifact, preventing duplicate signatures.

The Quorum System

The quorum requirement is enforced at the build system level, not through social convention. The publish-{stage}-{name} target in the Makefile, generated by src/targets.py, contains the following logic:

index_digest="$$(jq -r '.manifests[0].digest | split(":")[1]' out/{stage}-{name}/index.json)"
digest="$$(jq -r '.manifests[0].digest | split(":")[1]' out/{stage}-{name}/blobs/sha256/$${{index_digest}})"
signum="$$(ls -1 signatures/stagex/{stage}-{name}@sha256=$${{digest}} | wc -l)"
[ $${{signum}} -ge 2 ] || { echo "Error: Minimum signatures not met for {stage}-{name}"; exit 1; }

This code extracts the manifest digest from the built OCI layout, counts the number of signature files in the signatures repository directory for that digest, and exits with an error if fewer than two are present. The publish target depends on all individual publish-{stage}-{name} targets, so failing the quorum check for any single package blocks the entire release.

The quorum of two is not an arbitrary threshold. It is derived from the principle that no single maintainer should have unilateral authority over the distribution. With a minimum of two, a compromise requires either:

The simultaneous subversion of two maintainers' independent build environments and signing keys, or
Collusion between two maintainers, which must survive the public record of signed attestations and the project's review processes.

A threshold of one would be equivalent to the single-signer model used by most other distributions, which the xz backdoor incident demonstrated to be insufficient. A threshold higher than two would increase assurance but at the cost of release velocity; the project may raise this threshold as the maintainer base grows.

Key Management

StageX's key management requirements are specified in the MAINTENANCE.md document and the MAINTAINERS file. Every maintainer must adhere to the following:

Hardware security modules. All signing keys MUST reside on high-quality smart cards: YubiKey series 5, NitroKey 3, or equivalent. An alternative is a Split GPG Qubes OS setup. Software-only keys are explicitly prohibited.

PIN protection. Both user PIN and admin PIN MUST use non-default passwords. This prevents an attacker who gains physical access to a smart card from using it without the PIN.

Touch required. All PGP operations MUST require physical touch on the smart card. This prevents remote or automated signing -- an attacker who compromises the host system cannot initiate a signing operation without physical interaction.

Offline certificate authorities. Most maintainers use offline CAs. The MAINTAINERS file shows that 6 of 8 maintainers have offline-ca = true for their personal keys. This means the primary key is generated and stored on a machine that is never connected to a network; only subkeys (used for daily signing operations) reside on internet-connected machines.

No internet-connected key material. Private key material MUST never be exposed to an internet-connected environment. This eliminates the most common vector for key exfiltration.

Key diversity. Two maintainers use ED25519 keys and the remaining six use RSA 4096-bit keys. This diversity means that a cryptographic breakthrough affecting one algorithm does not compromise the entire signing infrastructure.

The eight current maintainers are distributed across North America and Europe, using a range of hardware: YubiKey 4 Nano, YubiKey 5 NFC, YubiKey 5C, FST-01, and NitroKey 3, running on operating systems including QubesOS, Debian, Arch Linux, and Fedora.

The Signature Workflow

The signing process is split between two scripts: sign-all.sh (orchestration) and sign.sh (per-package signing).

sign-all.sh is the entry point invoked by make sign. It performs the following steps:

Verifies the working tree is clean (no uncommitted files).
Clones or updates the signatures repository from https://codeberg.org/stagex/signatures.git (with an SSH push URL configured for write access).
Determines the release branch name from the version generator.
Checks out or creates the release branch in the signatures repository.
Resolves the maintainer's signing key fingerprint from git config user.signingkey.
Calls sign.sh for every digest in the digests/*.txt files.

sign.sh handles an individual package signature:

Extracts the index digest and manifest digest from the built OCI layout (out/<package>/index.json).
Creates the target directory in the signatures clone: signatures/stagex/<package>@sha256=<digest>/.
Checks whether the current maintainer's fingerprint already appears in any existing signature file in that directory (deduplication).
Constructs the JSON payload and signs it with GPG.
Writes the binary signature to an auto-incremented signature-<N> file.

The resulting directory structure looks like:

signatures/
  stagex/
    pallet-rust@sha256=2fbe7b164dd92edb9c1096152f6d27592d8a69b1b8eb2fc907b5fadea7d11668/
      signature-1    (PGP binary signature from maintainer A)
      signature-2    (PGP binary signature from maintainer B)

Both scripts use environment variables for the GPG toolchain: STAGEX_GPG, STAGEX_GPG_SIGN, and STAGEX_GPGV. This indirection allows maintainers to use custom wrappers, including Split GPG configurations or hardware proxy tools.

Verification

Users can verify signatures in three ways, at increasing levels of automation.

Direct GPG verification: Reconstruct the JSON payload manually and verify each signature-<N> file with gpg --verify. This is the most transparent method and does not require any specialized tooling beyond standard GnuPG. Each signature confirms that a specific maintainer independently built the package and confirms its digest.

Podman policy verification: Podman has built-in support for the Container Signature Format. By configuring ~/.config/containers/policy.json with a signedBy rule pointing to the StageX keyring and ~/.config/containers/registries.d/default.yaml with the lookaside URL at https://sigs.stagex.tools, Podman verifies signatures automatically on every pull. Images failing signature validation are rejected before their layers are downloaded.

Custom policies: Because signatures are stored in a standard format at a standard location, any tool that understands the Container Signature Format can verify them. Organizations can integrate StageX signature verification into Kubernetes admission controllers, CI/CD pipelines, or custom policy engines using the same sigs.stagex.tools lookaside server.

For detailed verification instructions, see the Verify Image tutorial and the Verify Attestations how-to guide.

Attack Resistance Matrix

The following table describes how the multi-signature system contains the blast radius of various compromise scenarios:

Attack Scenario	Effect Without Multi-Sig	Effect With StageX Multi-Sig
Maintainer A's signing key is compromised	Attacker can sign and publish arbitrary packages under A's maintained packages	Attacker has one signature, needs a second. Cannot publish without another maintainer's independent reproduction and signature.
Build infrastructure is compromised	Attacker can inject malicious code into build artifacts; users cannot detect the substitution	Attacker's modified artifact will not reproduce on another maintainer's machine. Digest mismatch detected. No second signature, no publication.
Source repository is compromised	Attacker can push arbitrary code changes	Changes require signed merge commits from 2+ maintainers. Single-committer changes are rejected by policy.
Registry is compromised	Attacker can replace images with malicious versions	Signatures are stored in a separate lookaside repository, not the registry. Replaced images fail digest verification against the independently stored signatures.
Multiple maintainers' keys are compromised	Collusion required, but possible if distribution has poor key diversity	Maintainers use diverse hardware (YubiKey, NitroKey), algorithms (RSA4096, ED25519), operating systems (QubesOS, Debian, Arch), and geographic locations. Simultaneous compromise is substantially more difficult than compromising a homogeneous infrastructure.
Cryptographic compromise of PGP	Attacker can forge signatures	Quorum of 2 provides defense-in-depth alongside key diversity (two different key types). Migration to post-quantum algorithms is a planned future improvement.

In every case, the attacker must overcome multiple independent barriers, none of which is under a single organization's control. This is the defining property of decentralized trust: the absence of a single point whose compromise leads to total system compromise.