Releases and provenance

A reviewed master commit can prepare a signed draft before its release tag exists. Publication requires a separate manual request from that tag naming the qualified manifest digest. The artifact set contains:

• a linux/amd64 and linux/arm64 manager/runner image;
• a reference to the Ptah release image the operator runs as its executor, the one support/ptah.json pins by digest at the selected source;
• a reproducible Helm chart asset whose version and appVersion match the tag;
• a keyless signature and GitHub build provenance for the image digest;
• GitHub build provenance for every downloadable asset;
• the reproducibly packaged chart, a digest manifest, and SHA-256 checksums;
• the acceptance evidence of the CI run that proved the candidate.

The release body is that digest manifest and nothing else, so what a version asks you to change before you move to it is Release notes rather than anything published on the release itself.

The release workflow refuses a mismatched tag, mutable external Docker image input, unpinned action, incomplete publish permission set, or manager image tag that differs from the chart version. It publishes no version or latest image alias.

A build is eligible for release only after the same commit passes the complete real-cluster lifecycle on every minor in the generated Kubernetes support window. Advancing that sliding window adds the new minor and removes the oldest minor in one reviewed change; publication must not substitute a preferred-minor smoke test for the required matrix.

Each matrix lifecycle installs a reproducibly packaged .tgz, rather than the chart source directory, and exports those exact bytes only after the complete upgrade and uninstall sequence succeeds. After the synthetic successor proof, the lifecycle fresh-installs that exact current-release package, verifies its controller identity and admission inventory, then proves every captured inventory object absent after uninstall before export. CI retains one artifact per supported minor for the completed run. Release preflight accepts only the complete, unexpired artifact set from the exact successful default-branch run, requires all copies to be byte-identical, and passes their SHA-256 digest into the protected publish job. The chart rebuilt from the selected commit must match that tested digest before any release asset can be materialized. A published-release claim still requires an actual immutable release; source CI proves package installability, not publication state.

The checksummed and attested release-manifest.txt records the selected support-evidence-run-id and the canonical kubernetes-support-window in addition to the tested chart digest. The identifier intentionally names the run-level evidence set: a GitHub rerun keeps its run ID, successful matrix jobs replace their same-name per-minor artifacts, and the aggregate gate accepts the run only after the latest job results succeed. A prepared journal is only transaction intent and does not claim a CI run; recovery may select another successful exact-source run before the final manifest is materialized. CI artifacts expire, so the pointer alone would preserve provenance but not the artifact bytes; the release keeps those in its acceptance evidence, described below. The installed charts expire after 90 days and the lifecycle timings after 30, so a candidate has to be released within 30 days of its CI run. If no complete unexpired exact-source evidence set remains, release or recovery preflight fails closed and requires a new release commit.

Prepare a release

Signed preparation uploads a draft and digest-addressed images without creating the Git tag or publishing the official release. Before preparing v0.2.6:

1. Merge the changes through a pull request, then wait for the complete CI run on that exact master commit. Record the commit and run URL. The pull request run alone cannot authorize publication.
2. Run go run ./hack/releaseverify -tag v0.2.6 at that commit. The chart version, appVersion, default image tag, and release notes must name the same version. Keep the installed-chart and lifecycle artifacts from its CI run; preflight checks their bytes and expiration.
3. Confirm immutable releases are enabled. The release environment must require a reviewer, accept only the exact master branch and v* tags, and contain the fine-grained IMMUTABLE_RELEASES_READ_TOKEN described below. A repository secret is not a substitute for the environment secret. Protect version tag creation and refuse updates and deletion.
4. Read the generated acceptance record with go run ./hack/acceptancecoverage -record -profile support/acceptance/lab-20.json. The profile remains Not assessed and excludes database restore/RTO and production capacity/soak qualification. A successful build does not close production qualification.

Prepare the signed assets from the recorded master commit:

gh workflow run release.yml --ref master -f action=prepare

The protected job checks that master still names that exact successful source commit and that the intended release tag is absent. The draft’s target_commitish is the full commit SHA. Its journal, signatures and attestations name refs/heads/master as the actual producer. GoReleaser receives the intended version explicitly and skips its tag-existence validation; the workflow checks the live source identity and a clean checkout independently. The job stops before publication and records the SHA-256 of release-manifest.txt. Keep that digest with the qualification results. Staged images are readable by digest; the draft does not establish an official immutable release.

Repeat the same preparation command to recover an interrupted transaction. An incomplete build must resume from its original producer ref and source. Preparation and publication share one concurrency group so they cannot mutate the same draft simultaneously. Unrelated pull request smoke runs remain separate.

After qualifying those exact assets, creating the release tag requires a separate decision. It must point to the manifest’s source commit. A tag push only prepares or re-verifies the draft; it does not publish the release. Request publication from that tag with the recorded manifest digest:

qualified_manifest_sha256='replace-with-the-recorded-64-character-digest'
gh workflow run release.yml --ref v0.2.6 \
  -f action=publish -f manifest_sha256="$qualified_manifest_sha256"

This run requires approval of the protected release environment again. It refuses a missing draft, an incomplete prepared journal or a different manifest before any build can start. It re-verifies the complete existing transaction, then compares the manifest digest again immediately before publishing. It does not replace the selected image digests or assets. A published immutable release remains a read-only verification state. A manual dispatch without an action still runs smoke checks. Preparation accepts master or the exact release tag; publication requires the exact release tag.

On the first publication, GHCR packages may not exist until the workflow pushes them. If either anonymous image check fails because its package is private, make that package public and rerun the same transaction as described below. Do not move the tag or replace a published artifact.

After publication, verify the immutable release and its assets, and run the documented installation and critical lifecycle checks using the published image digests and chart. Those checks require the published bytes; preparation and source CI cannot stand in for them. Once the tag exists, add its compatibility row to support/ptah.json with the executor pin and measured scope from that tag. Set documentation.published to true and documentation.source to v0.2.6 to build its guide from the tag. Do not declare a published guide before the tag exists.

Acceptance evidence

A CI run’s logs and artifacts expire; an immutable release does not. So the release keeps the evidence of the run that proved it as one more asset, acceptance-evidence.tar.gz, which holds:

• jobs.json: the CI run’s ID and attempt, and for every job the release requires, its name, job ID, the attempt that concluded it, and its conclusion, as the GitHub API reported them. The required jobs are the ones the Kubernetes support gate needs, one lifecycle per supported minor and suite, and the gate itself;
• artifacts/lifecycle-timings-<minor>-<suite>/: what each lifecycle uploaded, its timing ledger, run context, timing report and resource samples;
• acceptance-record.md: the output of go run ./hack/acceptancecoverage -record -profile support/acceptance/lab-20.json at the release commit.

The support preflight builds the bundle from the run it has just verified. It refuses the release when any required job did not conclude in success on the attempt that last ran it, whether it failed, was canceled or was skipped, and when any lifecycle’s download is missing or incomplete. The bundle is deterministic: entries sorted by name, one modification time (the release commit’s), root ownership, mode 0644, and a gzip header with no timestamp. The publish job takes exactly the bytes the preflight reported. It records their digest in the manifest as acceptance-evidence-sha256, lists them in SHA256SUMS, attests, uploads, and downloads them back like every other asset, and hack/releaseverify opens the bundle again and requires jobs.json to name the same run as support-evidence-run-id and a success for every required job.

Interrupted preparation rebuilds the bundle from the CI run the preflight selects. Once the manifest is final, that bundle has to match its recorded digest. Publication downloads the selected draft assets and authenticates their existing bytes; it does not rebuild its client binaries or evidence bundle. A published release keeps the bundle it published.

The executor

The executor is Ptah itself, the program an operation Job runs against the database. A release does not build it. It names the Ptah release image ghcr.io/stokaro/ptah at the digest the edge row of support/ptah.json pins, which is the image the acceptance suite copies and runs unchanged.

The workflow reads that image back from the registry before it writes the manifest, and refuses it unless the index digest is the pinned one and the index carries both linux/amd64 and linux/arm64. Ptah’s own release workflow builds, signs and attests the image; this release adds no signature of its own to it.

The release manifest records the image as executor and the build it carries as executor-ptah-commit and executor-ptah-version. The verifier refuses a manifest whose image, commit or version differs from the catalog at the tag.

The chart still has no executor default. The release names a build; the installer decides to pin it, and can pin another verified executor instead. Ptah compatibility says what the pinned build was run against.

Controller-state contract

The controller-state version says which durable status a manager can read. A manager reads state stamped at its own version or below, and the release fence refuses anything above it: an upgrade may move forward, a rollback may not move behind state that has already been written.

Version	What state at this version carries that an earlier manager cannot read
1	The durable contract of the first release: execution bindings, plan and applied records, approvals, the operation claim, status.unresolvedRun on a PtahMigration (the record of an Apply whose outcome nobody established), and status.activeOperation.retryNotBefore (the delay a retried operation waits out).

A resource records its version where the run is bound to the components that executed it, and never clears it. A record a resource can only hold after it has dispatched, as both of the last two are, needs no location of its own: dispatching is what writes status.executionBinding.controllerStateVersion.

During release preparation, the version was reset to 1 with the other contract counters. From the first release on, it moves only when a manager writes state that a manager of an earlier release could not read.

The number is declared in two places that this repository holds equal: CONTROLLER_STATE_VERSION in the Makefile, which stamps operator.ptah.run/controller-state-version into every shipped CRD, and controllerstate.CurrentVersion, which the manager compiles. make verify refuses a tree where the two disagree.

What a refusal looks like

The state preflight runs in the chart’s CRD hook on install, upgrade and rollback, before the hook stops the running release and before the first CRD update. A refusal fails the hook and leaves the running release as it was; the message names the resource and the location it read, for example:

controller downgrade refused: PtahMigration orders/billing stores controller
state version 2 at status.executionBinding.controllerStateVersion, but this
manager supports 1

Neighboring refusals come from the same fence and mean different things. A chart and a manager image from different releases are caught before any resource is read:

CRD ptahmigrations.operator.ptah.run annotation
operator.ptah.run/controller-state-version=1
does not match compiled controller-state version 2

Install the chart that was published with that image. A candidate whose CRDs predate the state contract the installed CRDs record is refused before any CRD is touched, whether it arrives as an upgrade or as a rollback:

controller-state rollback refused: existing operator.ptah.run/controller-state-version=2
is newer than candidate version 1

None of the three is recoverable by retrying. A cluster whose resources carry version n accepts a manager compiled for n or later, so the way forward is the release that reads the state already written, not an older one.

Publication transaction

A fresh transaction first creates and attests a minimal state=prepared journal that binds the intended version, producer ref, source commit, stable transaction ID, expected chart name, and the exact retention tag ghcr.io/stokaro/ptah-operator:tx-<source-sha>-<run-id>. It stores those exact bytes as the body of an empty draft release and verifies the body before any registry push. A rerun keeps the run ID recorded in that journal; the attempt number is deliberately not part of the transaction identity.

After the prepared draft exists, the workflow logs in to GHCR and inspects the exact retention tag. The inspection is authenticated because GHCR refuses an anonymous read of a package that does not exist yet exactly as it refuses a private one, so only an authenticated read tells a first release’s missing tag from a denial. Anonymous readability is proved separately before publication. An existing raw manifest is reused only when its exact digest has an authenticated build checkpoint from this release workflow, repository, producer ref, and source commit. That checkpoint is created solely from the digest returned by a successful image-build action. A missing or uncheckpointed tag is rebuilt; the replacement cannot become reusable until the new build output has its own checkpoint. The registry’s one-line missing response must name the exact tag that was inspected. An unavailable, multiline, or otherwise ambiguous response fails closed. A transaction tag is not a release identity and must never be consumed. The final authenticated manifest records the image digest beside the executor the catalog pins. The image includes a maximal BuildKit provenance record and SBOM. Chart packaging normalizes all source timestamps to the source commit time and CI requires two independent packages to be byte-identical.

The workflow attests release-manifest.txt, compares the prepared draft body, then replaces that body with the same authenticated final manifest bytes before synchronizing the release assets. It never replaces uploaded asset bytes. A failed upload may leave an empty starter asset; recovery deletes only that exact incomplete asset ID and uploads the journaled bytes again. Any uploaded mismatch, duplicate name, unexpected asset, or unknown state fails closed. Image signing, attestation, and anonymous digest pull verification all finish before the draft is published.

The build boundary is versioned as data. Actions use audited commit pins, Buildx uses an exact version, and its BuildKit daemon image is selected by multi-architecture digest with the action’s binary cache disabled. The BuildKit SBOM generator is also selected by digest rather than its mutable convenience tag. The manager binary is compiled in the digest-pinned Docker builder stage and runs on a digest-pinned non-root base. The publish job also disables the Go action cache; its small release verifiers and deterministic chart packager are rebuilt from the checked-out source and checksum-locked modules for every transaction.

Rerunning a partially completed release first authenticates either the prepared journal or final manifest against its exact producer ref, source commit, and signer workflow. The producer is restricted to refs/heads/master or the selected release tag; later publication preserves that original identity. From a prepared journal it reuses each existing staged digest only after verifying the exact build checkpoint described above; otherwise it rebuilds into the same transaction tag and checkpoints the action’s returned digest. It reproduces the chart bytes deterministically and resumes missing additive steps. A published immutable release is a read-only recovery state: the workflow re-authenticates its tag, body, assets, both image signatures, image attestations, and anonymous image availability, then succeeds without mutation. A published but mutable release, a moved source tag, a mismatched asset, or an unavailable state check fails closed. No step uses asset replacement.

Every mode, the read-only one included, reads the pinned Ptah image back from the registry, so a rerun also needs ghcr.io/stokaro/ptah to serve that digest.

Before publication, the image gate compares the transaction retention tag with the manifest-list digest recorded in the final release manifest. It requires exactly the linux/amd64 and linux/arm64 runtime manifests, validates source, revision, and version labels on each platform image, and binds each platform to its own SBOM and maximal BuildKit provenance attestation. The provenance must carry the release build arguments and detailed build graph. A single fail-closed parser enumerates external image inputs from case-insensitive Dockerfile instructions, line continuations, resolved ARG defaults, every FROM, every external COPY --from, and every external RUN --mount source. Unsupported parser constructs and unresolved or mutable references are rejected. The publish build arguments are structurally fixed so they cannot override an enumerated image reference. The pinned Dockerfile syntax frontend is an external input too. Every enumerated digest must occur as an exact SHA-256 value in each platform’s structured resolvedDependencies; merely placing a digest string elsewhere in the predicate cannot satisfy the gate. The same exact material check separately requires the pinned SBOM generator digest. This structural and material verification finishes before the final image attestation or signature is created. The same gate runs during published recovery.

Repository administrators must enable immutable releases before publishing the first version. This makes the published tag and assets platform-enforced immutable and adds a release attestation that binds the tag, commit, and asset digests. Drafts intentionally remain mutable so the transaction can recover; the workflow authenticates and compares their contents before publication. The workflow requires a fine-grained IMMUTABLE_RELEASES_READ_TOKEN with repository Administration read permission and checks the setting before pushing anything. Keep this token only in the protected release environment, never as a general repository secret. The final step requires both the API’s immutable flag and a valid release attestation.

Configure the release environment with required reviewers and restrict it to the exact master branch and v* release tags. Keep the required reviewer for both. Protect v* creation with a repository ruleset and require the tagged commit to have passed the default-branch review and CI policy. The workflow independently proves that the tagged commit is reachable from the current default branch. These repository controls are part of the release trust boundary: a workflow file loaded from an unreviewed tag must never receive the environment secret or OIDC publication authority.

Administrators must also make the image package ghcr.io/stokaro/ptah-operator public before the first release. The workflow logs out of GHCR and proves that the exact image digest is anonymously readable before publishing the draft. GHCR creates a package on its first push. If a first release stops at the anonymous check because the package is still private, make it public and rerun the workflow; the rerun resumes the same transaction.

Immediately before publishing, the workflow re-fetches the draft body and exact asset-name set, downloads every asset, compares its bytes with the locally attested transaction, re-verifies each build attestation, and re-validates the manifest/checksum/chart relation. GitHub does not expose an atomic compare-and-publish API for drafts, so repository write access remains a trusted administrative boundary during this short gate. Immutable-release enforcement locks the tag and bytes at publication, after which the workflow verifies the release attestation.

Verify before installation

Choose the tag independently, resolve its commit, and authenticate the release and its assets before trusting the manifest or its checksums. The recorded producer ref selects between the exact release tag and refs/heads/master; every other ref is refused, and attestations still require the resolved release commit and the exact signer workflow:

tag=v0.2.6
version=${tag#v}
repository=stokaro/ptah-operator
source_sha="$(gh api "repos/$repository/commits/$tag" --jq .sha)"

gh release verify "$tag" --repo "$repository"
gh release download "$tag" --repo "$repository" \
  --pattern "ptah-operator-$version.tgz" \
  --pattern release-manifest.txt \
  --pattern SHA256SUMS \
  --pattern acceptance-evidence.tar.gz

for asset in \
  "ptah-operator-$version.tgz" \
  release-manifest.txt \
  SHA256SUMS \
  acceptance-evidence.tar.gz
do
  gh release verify-asset "$tag" "$asset" --repo "$repository"
done

source_ref="$(sed -n 's/^source-ref=//p' release-manifest.txt)"
case "$source_ref" in
  "refs/tags/$tag"|refs/heads/master) ;;
  *) echo 'unexpected release producer ref' >&2; exit 1 ;;
esac
for asset in \
  "ptah-operator-$version.tgz" \
  release-manifest.txt \
  SHA256SUMS \
  acceptance-evidence.tar.gz
do
  gh attestation verify "$asset" \
    --repo "$repository" \
    --source-ref "$source_ref" \
    --source-digest "$source_sha" \
    --signer-workflow "$repository/.github/workflows/release.yml"
done

grep -Fx "version=$version" release-manifest.txt
grep -Fx "source-repository=$repository" release-manifest.txt
grep -Fx "source-ref=$source_ref" release-manifest.txt
grep -Fx "source-sha=$source_sha" release-manifest.txt
sha256sum --check --ignore-missing SHA256SUMS
grep -Fx "acceptance-evidence-sha256=$(sha256sum acceptance-evidence.tar.gz | awk '{print $1}')" \
  release-manifest.txt

The acceptance evidence is then readable as it was when the release was cut:

tar -tzf acceptance-evidence.tar.gz
tar -xzOf acceptance-evidence.tar.gz jobs.json |
  jq -r '"run \(.runId) attempt \(.runAttempt)", (.jobs[] | "\(.name)\t\(.id)\t\(.attempt)\t\(.conclusion)")'

Only after those checks should the image reference be read from the manifest. Verify its provenance against the same producer ref and release commit:

image="$(sed -n 's/^image=//p' release-manifest.txt)"

gh attestation verify "oci://$image" \
  --repo "$repository" \
  --source-ref "$source_ref" \
  --source-digest "$source_sha" \
  --signer-workflow "$repository/.github/workflows/release.yml"

Cosign provides an independent signature check. The release signs with Cosign v3, which stores the signature as a Sigstore bundle, so verify with Cosign v3 or later: Cosign v2.5 finds no signature on the image. Bind its certificate identity to the authenticated producer ref:

identity="https://github.com/$repository/.github/workflows/release.yml@$source_ref"
cosign verify \
  --certificate-identity "$identity" \
  --certificate-oidc-issuer='https://token.actions.githubusercontent.com' \
  "$image"

The executor is the Ptah release image named in the same manifest. Its digest is executor, and the Ptah build it carries is executor-ptah-commit and executor-ptah-version. The manifest’s attestation above is what binds that digest to this release, so install exactly the digest it names:

executor="$(sed -n 's/^executor=//p' release-manifest.txt)"
ptah_version="$(sed -n 's/^executor-ptah-version=//p' release-manifest.txt)"

Install by digest

The chart asset, manager/runner image, and Ptah executor are all pinned separately. A release names an executor, and the chart takes it only as an explicit value. That lets an operator release be promoted without silently changing the database executable, and lets an installation pin another verified executor with the same operator release.

helm upgrade --install ptah-operator \
  "./ptah-operator-$version.tgz" \
  --namespace ptah-system \
  --create-namespace \
  --set-string "image.digest=${image##*@}" \
  --set-string "execution.runnerImage=$image" \
  --set-string "execution.executorImage=$executor" \
  --set-string "execution.ptahVersion=$ptah_version"

From v0.2.5 the image is also on Docker Hub, as docker.io/stokaro/ptah-operator:<version>, for a cluster that pulls without signing in. .github/workflows/mirror.yml copies it there without changing a byte once the release is published, so it is the same digest, and the attestations above, which are keyed by digest, verify it under either name. To pull from Docker Hub, set image.repository to docker.io/stokaro/ptah-operator and give execution.runnerImage as docker.io/stokaro/ptah-operator@ followed by the same digest. Docker Hub limits anonymous pulls; GHCR does not.

execution.ptahVersion has no chart default and is not inferred from the image reference. Set it to the version identity established by the provenance of the exact executor digest. For the executor a release publishes, that is executor-ptah-version in the authenticated manifest. For any other executor, it is whatever that image’s own provenance establishes. The operator records that pair in plans, approvals, operation Jobs, and applied status, so an executor change is an explicit new execution binding even when the operator release is unchanged. Release builds also inject the exact manager source revision. The manager refuses to start without both a digest-pinned manager image identity and that revision, and records them on every plan it publishes and every Job it dispatches. They are a record, not a binding.

Changing execution.ptahVersion or execution.executorImage, or rolling out a release with a different controller-state contract or runner protocol, intentionally invalidates a plan and recorded approval until a mutating Job has been dispatched. A claimed but undispatched Apply also returns to read-only reconciliation. Wait for the replacement manager to finish Resolve, Verify, Observe, and Plan, review the new plan UID and fingerprint, and issue a new approval. Do not carry approval objects across an execution-binding upgrade as deployment automation. A dispatched Apply remains bound to its captured execution identity and proceeds through conservative outcome classification and post-Apply observation; the upgrade never recreates it with different binaries.

A release that changes only the manager image, the manager revision and execution.runnerImage – which the release builds together, from one source tree – invalidates nothing. The execution epoch, the current plan and any pending approval carry over, and the replacement manager applies the approval. It adopts a Job its predecessor dispatched only when it builds the same Job apart from the recorded manager identity. A release that also changes the Job or its Pod template settles a dispatched PtahSchema Apply as outcome unknown, which a read-only observation then resolves, records a dispatched PtahMigration Apply as an Unknown run, which a History reading or an acknowledgment then settles, and runs a dispatched read-only Job of either family again. Normal status progression states the rule. The runner’s enforcement is bound through the runner protocol version rather than its digest, and a release that changes what the runner enforces bumps that version.

Two checks hold that rule. make verify-source runs hack/verifyrunnerprotocol, which digests the source the runner is built from – every non-test Go file of this module that ./cmd/ptah-runner imports, and the go.sum lines of every other module it imports – and refuses a digest that moved while runner.ProtocolVersion did not. A change that leaves the runner’s contract as it was is declared in support/runner-protocol.json with its reason, under the protocol version it was made in. At run time, every Job names the protocol its manager speaks in PTAH_RUNNER_PROTOCOL_VERSION, and a runner that speaks another one – an execution.runnerImage from another release – refuses the Job before the executor starts.

status.executionBinding exposes the controller-state contract, Ptah version, executor image, runner protocol, and its opaque epoch for audit. Every transition of those components creates a new epoch, including rollback to an identical set. Plan contract 1 binds exactly those, and records the publishing manager on the plan without binding it. A plan references the epoch as spec.executionBindingID and binds it into the fingerprint an approval names; therefore an approval is valid for only one transition and cannot be reused after rollout or rollback.

The invalidation boundary starts when the replacement manager owns reconciliation, not when helm upgrade is invoked. The Recreate strategy terminates every old manager Pod before starting replacement Pods, preventing mixed admission contracts behind the webhook Service. The old leader may still dispatch a fully old-binding approval before termination. For a strict no-dispatch maintenance boundary, scale the manager Deployment to zero and wait for all manager Pods to terminate before running helm upgrade --wait. The chart then restores its configured replica count.

For an air-gapped promotion, carry the authenticated chart asset and recursively copy the operator image, executor image, and every referenced schema artifact by digest into the destination registry. Record the source and destination digest mapping, then update only repository names in deployment values and PtahSchema references. A promotion that changes a digest is a rebuild and must go through verification again.