Operations

Start here

This page is long because the contracts are. If something is wrong now, the row that matches is where to go.

What you are looking at	Where it is answered
A first installation	Install, then Install the operator
An upgrade to run	Upgrade to a new release. A candidate refused before any CRD is touched leaves the running release in place.
An upgrade that stopped partway	Re-run an upgrade that stopped partway
A return to an earlier revision	Roll back to an earlier revision
A release with neither runtime Deployment left	Repair a release that lost its runtime
Removing the operator from a cluster	Uninstall the release
A namespace stuck in Terminating, or tearing one down in CI or a lab	Deleting a namespace
A resource that is not converging, and no obvious refusal	Finding a resource that has stopped converging
A resource in Blocked, or a condition you do not recognize	Condition reasons
A migration Blocked with HistoryDirty	A migration that failed at a statement
A migration whose Apply ended Partial or Unknown	A migration run nobody accounted for
A schema Blocked with TargetIdentityChanged after its URL Secret moved	A schema whose target moved
A new artifact to roll out	Moving to a new artifact
Writes failing because the webhook cannot be reached	Webhook certificate lifecycle
A write the admission refused, with a message	The message names the guard; Security model says what each one protects
The cluster or the namespace is gone and the database is not	Recover operator state
Storage growing with every plan	Plan retention, then Pruning stored plans
Whether the operator can run SQL nobody approved	Execution guarantees

When a procedure does not reach the outcome it states, stop instead of improvising a cleanup this page does not describe, and escalate. First go to whoever owns the installation in your organization. Then go to the project: open an issue at https://github.com/stokaro/ptah-operator/issues with the resource’s YAML, its Events and the procedure you followed, with database URLs and other credentials removed.

Before anything else, do not delete a PtahMigration to clear a state you do not like. Its record of an Apply nobody accounted for goes with it, and that record is what a person needs to find out whether the database was changed.

Installation and upgrades

Every task here names what has to be true before it starts, the commands, what proves it finished, where to stop, and what to do when it fails. The hooks check a great deal more than any of these tasks say, and what each refusal means is Release lifecycle.

Helm 4 or newer is required, and every lifecycle contour is verified against it. Helm 3 is not supported: it reaches end of life before this operator’s first release, and it applies client-side, so the release’s objects carry no server-side apply ownership for a later upgrade to take back.

Install the operator

Install carries a first installation end to end: the values, the digests, and the readings that settle whether the manager is serving. What follows is what any installation has to satisfy, including the ones a first install never has to think about.

Before you start

You need cluster-admin: the CRDs, the admission policies and their bindings are all cluster-scoped.

Install exactly one Helm release of the operator in a cluster. The manager watches cluster-wide resources, and admission uses the singleton ptah-operator-admission webhook configurations, so a second release would create an independent availability domain capable of blocking the first. The fixed configuration names make an ordinary second Helm install fail ownership validation. High availability is replicaCount within the one release, with leader election enabled. Helm rejects more than one replica when leaderElection=false; disabling election is supported only for an isolated cluster with one operator release and one replica. The manager Deployment uses Recreate for both modes. Every ready replica serves admission traffic even when it does not hold the reconciliation Lease, so an old and a new revision must never overlap behind the webhook Service.

The leader holds its Lease for 60 seconds and renews it within 45, so an API server that stalls for less than that does not restart it. A manager that stops releases the Lease once its reconcilers have returned, within 50 seconds and so inside the Pod’s 60-second grace period: a restart or an upgrade hands over at once, and only a manager that dies holding the Lease waits for it to run out, about a minute. A stopping manager starts no new Job, and one whose dispatch had already begun finishes creating it, so the next leader adopts that Job instead of recording a run nobody accounted for. Only such a dispatch, with a slow Job-write webhook, makes the stop take that long.

Install with digest-pinned manager, executor and runner images. The chart a release publishes carries them, and any chart refuses all three when only a tag is supplied. There is no local-tag mode: image.allowMutableTag and image.testIdentityDigest are not chart values, so a values file that still names either fails schema validation, and a render that skips schema validation refuses them by name. Manager Pods, hooks and controller identity all use the same image.repository@image.digest reference.

Install into a dedicated namespace and keep it, for the life of the release, to principals you trust to administer Ptah. The release namespace, and coordination.namespace when it names another one, is a privileged administrative boundary: Kubernetes lets whoever can create a Pod in a namespace run it as any ServiceAccount there, so anyone who can create or modify workloads, exec into Pods or request ServiceAccount tokens in it can act as the operator, and is a Ptah administrator. Deploy no application workloads there, and grant no namespace-admin, edit or workload-creation access to anyone else. Security model is the contract, and it holds for installs, upgrades and uninstalls alike. Cluster administrators and principals that can change admission policy are inside the same boundary. When Helm can read the cluster, the chart refuses default, a kube-* namespace, or a namespace that runs workloads without this release’s app.kubernetes.io/instance label, and its notes warn about RoleBindings that make somebody else an administrator there. What the chart checks is the whole list, and what a render without the cluster skips.

The controller runs as one ServiceAccount in every release. With serviceAccount.create=false, create the ServiceAccount serviceAccount.name names in the release namespace before the first install; every later release runs as the same one. The chart binds its roles to it and never deletes it.

For deterministic GitOps rendering, provision the webhook TLS Secret outside the chart and set both webhook.existingSecret and the PEM-encoded webhook.caBundle. A connected Helm install can instead reuse ca.crt from an existing Secret. Setting webhook.existingSecret completely disables the built-in certificate lifecycle resources, so disabling certificateRotation.enabled requires it; the chart refuses an unmanaged generated certificate. Moving a release from the built-in rotator to a Secret of its own needs delivery off and --force-conflicts on the upgrade, because the rotator’s field manager holds each webhook’s caBundle. A first interactive install can instead generate a self-signed Secret and its built-in rotation Deployment. Argo CD and Flux should depend on the CRDs and an externally managed TLS Secret before synchronizing the Deployment and webhook configurations.

For local development, push the built image to a registry every cluster node can reach and use that registry’s manifest digest in image.digest, with image.repository naming its repository. A Docker image ID is not a manifest digest. For kind, its local registry setup configures node access. Where the registry needs authentication, create the release namespace and the image pull Secret before running Helm and select the Secret through imagePullSecrets, so installation hooks can pull their image too.

Run it

helm install <release> <chart> --values <values>

Installing over CRDs an earlier release left behind needs --force-conflicts when anything else has edited them. Helm applies the chart’s CRDs server-side on install, so a field a kubectl patch or another controller owns is a conflict until the install is told to take it back:

helm install <release> <chart> --values <values> --force-conflicts

The flag makes the chart’s CRDs win over whoever edited them. Use it when that is what you mean.

What proves it worked

Helm reporting success means its hook completed, which is not the same as the manager serving. Confirm it installed carries the two readings that settle it: ten CRDs at Established=True, and the manager and certificate-rotator Deployments available.

Where to stop

A second release in the same cluster is not a supported configuration, and neither is a values file reaching for a mutable tag. Both are refused rather than warned about, and there is no value that accepts them. An installation whose CRDs carry no schema identity is a third: the operator intentionally provides no value that labels an unknown schema as trusted.

A release namespace the chart would share is refused as well, and that one has a value: releaseNamespace.allowSharedNamespace: true. Setting it says that everyone who can create workloads in the namespace administers Ptah. If that is not true, move the release, or the other workloads, instead.

If it fails

A refused CRD hook leaves the runtime unchanged, and the refusal it reports is the whole problem. Correct that and rerun the same command. Helm deletes the hook Job when it fails as well as when it succeeds, and prints its log to stderr before it does, so the refusal is in the output of the command that failed.

Upgrade to a new release

Upgrades are supported from the first published release onward.

Before you start

You need cluster-admin.

The release namespace check runs on every upgrade, so a workload somebody deployed into the release namespace since the last one refuses it until the workload leaves.

Every release since the first stamps the CRDs with the schema version, schema digest and controller-state version, and the admission singletons with their release identity. An installation that carries none of that is not an upgrade source: the chart refuses the upgrade before any change, and Offline singleton migration is the way forward.

Changing an established coordination namespace or leader-election mode is not an upgrade either. Both are pinned by the admission singletons, and connected Helm rendering fails when the requested values disagree with what they record. That is the same offline migration.

Free the quota the candidate Pods need. An upgrade that changes the manager image stops the running release in its hook, before Helm applies the candidate, so a namespace short of quota leaves no manager running until the candidate Pods fit. Nothing checks the quota for you.

Run it

An upgrade that changes the manager image needs --force-conflicts:

helm upgrade <release> <chart> --values <values> --force-conflicts

Such an upgrade stops the runtime in its pre-upgrade hook, which leaves .spec.replicas at zero under the hook’s own field manager, and the apply that follows has to raise it again. Server-side apply reports that as a conflict. The hook writes no other field, so the force is confined to the replica count the stop moved. An upgrade that keeps the manager image does not stop the runtime and does not need the flag:

helm upgrade <release> <chart> --values <values>

What proves it worked

The readings are the ones an install ends with, against the new digests: Established=True on ten CRDs, both Deployments available, and manager Pods whose image is the candidate’s.

An upgrade that keeps the manager image, the shape a GitOps re-sync or a values-only change produces, leaves the runtime running. The reconcile hook checks the stored state and the CRDs as on any upgrade, finds both runtime Deployments already on its image, and stops nothing. Helm then applies the manifests, and a changed value rolls the Deployments the ordinary way.

Where to stop

Do not edit the remaining CRDs to imitate the candidate, do not force server-side apply conflicts on them, and do not change singleton annotations to make an upgrade pass. Select a manager version compatible with the schemas already stored rather than working around the refusal.

A Helm --wait timeout while the namespace is short of quota is a capacity incident. The Deployment controller retries candidate Pod creation once the quota is free, and the manager serves again when it does.

If it fails

A refusal in the CRD hook before it stops the runtime leaves the runtime unchanged. Correct the reported conflict or API reachability problem and rerun the identical candidate chart, image and values.

A hook that reports the chart and the manager image come from different releases refused before it read the cluster. --reuse-values keeps a previous image.digest you set under a new chart; set image.digest to the one published with the chart and rerun. A release installed from the published chart with no values has none to keep, so its next chart brings its own images.

A CRD that lacks the schema version or the schema digest is refused before any CRD mutation, even when its live normalized spec matches the candidate exactly. A malformed annotation, an incomplete identity plus any schema difference, and a same-version digest collision are refused the same way. For such an installation, keep the managers offline and restore the identity the release that created the CRD stamped on it, or reinstall from the first published release after backing up every CRD and custom resource.

An interruption after the hook stopped the runtime is a different situation, and Re-run an upgrade that stopped partway is the runbook for it.

Re-run an upgrade that stopped partway

CRD updates are necessarily separate Kubernetes API transactions. The complete dry-run prevents predictable partial upgrades, but an API failure or a concurrent administrator change can still interrupt the real update sequence. An upgrade can also fail after its hook finished: the hook stopped the runtime, and then Helm’s apply of the candidate was refused or timed out.

Before you start

You need cluster-admin, as for the upgrade this is resuming.

Resolve the API or policy failure that interrupted the upgrade. No step of this runbook makes progress while it stands.

Find out how far it got. Until the reconcile hook stops the runtime, the predecessor keeps running. After the stop, both Deployments sit at zero replicas on the predecessor’s Pod template until Helm applies the candidate’s. Do not bring the predecessor back by scaling its Deployments up: its CRDs may already be the candidate’s, and its own init verifier then holds it.

Have the identical candidate to hand.

Run it

Rerun the identical candidate chart, image and values:

helm upgrade <release> <chart> --values <values> --force-conflicts

The hook checks the stored state and the CRDs again, finds the runtime already stopped, which changes nothing, and finishes any CRD update the interruption left behind. Helm then applies the candidate over the stopped Deployments.

What proves it worked

The same readings an upgrade ends with: Established=True on ten CRDs, both Deployments available, and manager Pods carrying the candidate image.

Where to stop

Do not scale the stopped Deployments up by hand to get the predecessor back. Roll back to an earlier revision is the supported way to the release before, and it runs the checks an upgrade runs.

If it fails

The refusal names what refused, and Release lifecycle says what the hook proves and therefore what has to change. Correct that and rerun the same candidate again; each attempt revalidates live state, so repeating it costs nothing and changes nothing on its own.

Roll back to an earlier revision

helm rollback restores an earlier revision’s manifests and runs that revision’s CRD hook first, with that revision’s image. The hook holds the rollback to the checks an upgrade passes: it refuses stored state and CRD schemas newer than the restored release reads. Every check but one runs before it stops anything; the last state scan runs after the stop, to find state the running manager wrote in the moment before it stopped.

Before you start

You need cluster-admin.

Rolling back below state already written is refused, whatever revision the history offers. The controller-state contract says which release reads which state.

Run it

helm rollback <release> <revision> --force-conflicts --wait --timeout 5m

The flag is needed for the reason an upgrade that changes the manager image needs it: the hook stops the runtime by scaling both Deployments to zero, and Helm raises the replica count again.

What proves it worked

helm status reports the new revision deployed, and both Deployments are available with manager Pods carrying the restored revision’s image.

Where to stop

Do not go back with kubectl rollout undo on the manager’s Deployment. It passes none of these checks and leaves the newer CRDs in place, and a manager writes status as a JSON merge patch of the fields it knows, so whatever nested field it does not know stays as it was. A manager older than 0.2.10 that records a run leaves the operationID, planRef, planFingerprint, authorization, source, targetIdentityDigest, executionBindingID and verification of the run before it on status.lastRun, beside its own Job and outcome. Read those as the earlier run’s, including where kubectl ptah approve --wait and kubectl ptah wait --plan read them, until a manager that knows the fields records the next run and replaces them.

Do not edit the stored state or the CRD annotations to let a refused rollback through. The refusal says the cluster holds state the older release cannot read, and the way forward is a release that reads it.

If it fails

Helm records the rollback revision before it runs the hook, so a refused rollback leaves that revision pending-rollback, and a later helm upgrade refuses to start while it is. A refusal before the stop leaves the running release as it was; one from the scan after the stop leaves the runtime stopped until the next rollback or upgrade brings a release up. Another helm rollback, to a revision the stored state allows, clears the pending revision.

helm upgrade --rollback-on-failure rolls back to the last deployed revision when the upgrade fails. After an upgrade that failed once its hook had updated the CRDs, that rollback runs the previous release’s hook, which refuses the newer schemas, and the release is left pending-rollback. helm rollback <release> with no revision rolls back to the revision before the pending one, which is the upgrade that failed: its hook reads the schemas it wrote, and Helm brings that release up.

Repair a release that lost its runtime

A GitOps prune or a namespace-wide delete that spared the release’s other objects can leave a release with neither runtime Deployment.

Before you start

You need cluster-admin, as for any upgrade.

Find the chart version that is installed, and use that one, so the repair changes nothing but the missing Deployments. An upgrade, if you want one, is the second step.

Run it

helm upgrade <release> <chart-at-the-installed-version> --values <values>

What proves it worked

Helm recreates both Deployments. The reconcile hook finds neither to stop and checks the stored state and the CRDs as on any upgrade. After that, the readings an install ends with.

Where to stop

Do not repair with a newer chart, and do not recreate the Deployments by hand. Restore the release at its installed version first and upgrade afterwards.

If it fails

A refusal here is the hook’s, and Release lifecycle says what it checks. Where the admission singletons no longer carry this release’s identity, the release is not repairable in place and Offline singleton migration is the path.

Uninstall the release

An uninstall is Helm deleting what the release installed, the admission policies and their bindings among them. No hook runs, and the CRDs stay.

Before you start

You need cluster-admin. The uninstall deletes cluster-scoped admission policies and their bindings.

Back up the CRDs and their custom resources. Helm retains both, and an uninstall removes the controller and admission resources rather than the database changes Ptah previously executed, but a backup is what makes the next install a decision rather than a hope.

A ServiceAccount you created is never deleted by the chart, and any credential or grant issued outside Kubernetes RBAC has to be retired by hand.

With the release gone, nothing refuses the deletion of a PtahResultRecord in any namespace, whether or not its retention window has ended.

Run it

helm uninstall <release> --wait --timeout 5m

What proves it worked

Helm reports the release uninstalled. What remains afterwards is deliberate: the ten CRDs with their custom resources.

Where to stop

Do not delete the CRDs to finish the job. Deleting a CRD deletes every resource of its kind, plans and approvals included.

If it fails

Helm names the object it could not delete. Correct that and rerun the same helm uninstall; Helm deletes what is left.

Offline singleton migration

This is the way past an installation the chart will not adopt, and the way to change a value the admission singletons pin.

Before you start

You need cluster-admin, write access to the release namespace, and a maintenance window on every database the suspended resources address.

Do not change singleton annotations merely to make an online upgrade pass. An installation whose ptah-operator-admission configurations carry no release identity predates the first published release, and the chart does not adopt it.

Changing an established coordination namespace or leader-election mode is the same procedure. Record the old coordination Leases before starting, and retain them as audit evidence until no interrupted operation can refer to them.

A migration suspended mid-sequence keeps status.history and, if one was left, status.unresolvedRun. The first is what says how far the sequence got, and the second is the record that stops a replacement Apply from running a migration that may already have executed. The record is the one that has to survive: the history is read again, and the record cannot be. The manager keeps a copy of it in the resource’s operator.ptah.run/unresolved-run annotation, so a backup that keeps metadata keeps the record even where it drops status; see A restore that drops status.

Run it

In this order:

1. Suspend every PtahSchema and every PtahMigration.
2. Wait for every operation Job of both families to finish.
3. Place the databases in a maintenance window.
4. Scale the manager and certificate-rotation Deployments to zero.
5. Back up all Ptah custom resources.
6. Uninstall the release, and verify that the ten CRDs and their objects remain.
7. Install exactly one release of the first published version or newer, with the new invariant values where those are what is changing.
8. Verify its CRDs, its admission annotations and manager readiness.
9. End maintenance and resume both kinds.

What proves it worked

The ten CRDs and their objects present after the uninstall, before anything is installed over them. Then the new release’s admission annotations carrying its own identity, manager readiness, and both kinds converging again.

Where to stop

Do not start with a migration still running: a migration left running is a writer this procedure does not stop. Do not install over the uninstalled release if the ten CRDs or their objects did not survive it, and do not treat a resource whose status.unresolvedRun and its copy in the operator.ptah.run/unresolved-run annotation both went missing as one that has nothing outstanding.

If it fails

Nothing here is time-bounded, so a failed step is repeated rather than worked around: the databases are in maintenance and both kinds are suspended, which is the state the procedure is safe to sit in. Restore the backed-up custom resources into the nine retained CRDs before resuming either kind.

coordination.namespace contains the fixed manager leader-election Lease and the database target Leases. It defaults to the release namespace and may name a separately administered namespace for the same release. Every resource that can mutate one physical database must claim the same realm, even when connection URLs use different aliases, proxies, or credentials: the same spec.target.coordinationKey when they all live in one namespace, or the same PtahRealm when they do not. See A database more than one namespace manages.

One database, one manager

Serialization is not ownership. Two resources that claim one realm never run at the same time, and they can still undo each other’s work by taking turns: one converges the database to a declared schema, the other applies a migration that changes it back, and each reports success.

So a database more than one resource claims is refused rather than queued. Every claimant goes Blocked with reason RealmConflict, no Job runs, and the status says how many resources claim the database and how many of them have not declared the claim shared.

Sharing one database between a PtahSchema and a PtahMigration, or between two of either, is a statement each of them makes for itself:

spec:
  target:
    coordinationKey: prod/application/orders-primary
    sharedRealm: true

One claimant that leaves it false blocks all of them, itself included. That is what makes the declaration mutual without any resource naming another: no resource can widen its own permission alone.

The operator verifies the declaration, not the disjointness. Nothing can tell whether two sets of arbitrary SQL touch the same rows, so sharedRealm: true means a person decided the areas do not overlap.

A resource that runs nothing claims nothing. Deleting one leaves the realm, and so does spec.suspend: true, which is how a resource steps aside without being deleted — the usual way to hand one database to the other claimant without declaring anything shared. Resuming it puts it back, and the refusal returns before any Job runs.

A declaration on one claimant does not wake the others, so a contested realm is re-examined on a bounded cadence: the shorter of the resource’s spec.interval and one minute. Every claimant leaves Blocked within a minute of the last one declaring, whatever interval it runs on.

A database more than one namespace manages

A coordination key belongs to its namespace. The realm it names is the engine, the namespace and the key together, so the same key written in another namespace is another realm, with a census and a Lease of its own. A key is a string anybody who can create a resource may write, and if it reached across namespaces, one resource created elsewhere with the key a tenant uses would put every resource of that tenant in RealmConflict, behind a status that names counts and cannot say where the claimant is.

A database that resources in more than one namespace manage is named by a PtahRealm, which a cluster administrator creates and which lists the namespaces it admits:

apiVersion: operator.ptah.run/v1alpha1
kind: PtahRealm
metadata:
  name: production-orders-primary
spec:
  engine: PostgreSQL
  namespaces:
    - application
    - reporting
  sharing: Shared

Each resource names it in place of a key:

spec:
  target:
    engine: PostgreSQL
    realmRef:
      name: production-orders-primary
    sharedRealm: true

The realm is the authorization. A resource that names it from a namespace it does not list, with another engine, or when no such realm exists goes Blocked with reason RealmNotAuthorized and runs no Job. It is also left out of the census of the resources the realm does list, so it blocks none of them. The refusal reads the same in all three cases, and names only the realm the resource asked for and its own namespace.

Among the listed namespaces the rules above hold unchanged, and the realm adds one of its own. sharing: Exclusive admits one claimant at a time, whatever each declares; sharing: Shared admits several when every one of them sets sharedRealm. The administrator permits the sharing and each claimant states that it manages only part of the database, and neither statement stands in for the other.

There is no default realm, and no realm is needed for a key. A key can reach only its own namespace, so it needs nobody’s grant; a realmRef can reach any namespace, so it needs one. A resource that names a realm nobody created is refused rather than treated as a key, because falling back would make the realm’s absence a way into a database that the realm, once created, would not admit.

The census reads the realm from the manager’s cache before every claim, and a change to a realm wakes the resources that name it. What the wake-up does depends on the direction. Removing a namespace refuses an idle resource there on the pass the change starts, and does not stop an operation already running: that resource is refused at its next claim, the same way a conflict is. Adding a namespace lifts a standing refusal at the refusal’s own re-check, which is at most a minute away, the same bound a peer’s declaration has.

A namespace name in a realm is a grant to whoever holds a namespace of that name. Remove a namespace from every realm that lists it before deleting it, because whoever creates a namespace of that name next inherits the grant. Where tenants choose their own namespace names – OpenShift project self-provisioning, Capsule tenants, HNC subnamespaces – do not list a name before its namespace exists either: the first tenant to create it claims the database, and can contest it for the namespaces the realm meant to admit, which is the refusal realms exist to withhold.

Only an administrator should be able to write a realm. The chart grants the manager get, list and watch on ptahrealms and nothing more, and examples/realm-administrator-role.yaml is a ClusterRole for the people who decide realm membership. None of the author, approver or diagnostic examples reaches realms.

A realm lists namespaces, so it is an administrator’s object: the operator writes no status to it, and nothing a tenant reads names another namespace. An administrator finds the claimants of one realm with a cluster-wide read:

kubectl get ptahschemas,ptahmigrations -A -o json |
  jq -r '.items[] | select(.spec.target.realmRef.name == "production-orders-primary")
    | "\(.kind) \(.metadata.namespace)/\(.metadata.name) \(.status.phase)"'

The manager has exact get, create, and update access to Leases through one namespace Role in coordination.namespace. When that namespace differs from the release namespace, its RoleBinding names the manager ServiceAccount across namespaces. The manager has no cluster-wide Lease rule and no Lease access in unrelated namespaces. Certificate rotation uses a different ServiceAccount and its exact release-namespace Lease grant.

Operation Pod memory

The runner and Ptah share one container in every operation Pod, so spec.execution.resources, or a LimitRange default where it is unset, bounds them together. Ptah v0.14.0 needs about twice the memory v0.13.0 did: for the small schemas of the acceptance suite, Observe and Plan take between 24 and 32 MiB on their own, where v0.13.0 took less than 16 MiB. A 64Mi limit that v0.13.0 fit in kills v0.14.0’s Observe and Plan Pods; the Pod reads OOMKilled, and the resource reports that the runner’s result frame was not found. The acceptance suite gives the container 128Mi.

Kubernetes admission configuration

The operator persists a bounded snapshot of built-in Pod admission before it dispatches an operation Job. Its final Pod validating webhook accepts the exact snapshotted LimitRange resource defaults, ServiceAccount image pull secrets, RuntimeClass scheduling and overhead, PriorityClass values, and the configured admission-plugin mutations. It rejects other changes to resources, tolerations, image pull policy, command, image, environment, volumes, or security settings before the Pod can be scheduled.

For regular and init containers, Kubernetes copies an admitted limit into a previously absent request after admission. The envelope accepts that API default only when the request is exactly equal to the separately validated limit. LimitRange API defaulting derives a missing container default limit from max, then derives a missing default request from that limit, and finally from min. The snapshot records those derived values explicitly. If more than one bound LimitRange offers a different value for the same unset field, the controller rejects dispatch instead of depending on list-order-dependent first-wins behavior. At most 64 request and limit default entries are retained across the complete UID- and resourceVersion-bound set.

The admission chart values must match the kube-apiserver admission chain:

• defaultTolerationsEnabled defaults to true. When enabled, defaultNotReadyTolerationSeconds and defaultUnreachableTolerationSeconds default to 300 and must match the API server flags. Zero means an exact zero-second toleration; it does not disable the plugin. When disabled, both automatically injected tolerations must be absent.
• extendedResourceTolerationEnabled defaults to false. Enable it only when kube-apiserver enables ExtendedResourceToleration.
• alwaysPullImagesEnabled defaults to false. Enable it only when kube-apiserver enables AlwaysPullImages.

A mismatch fails closed by rejecting the operation Pod. Updating a referenced LimitRange, ServiceAccount, RuntimeClass, or PriorityClass after snapshot persistence also rejects newly mutated Pods; retry reconciliation after the desired admission objects are stable. The manager receives only read access to those credential-free objects. It receives neither Secret read permission nor Pod create or delete permission.

The webhook deliberately does not model PodNodeSelector or PodTolerationRestriction defaults; clusters that inject either into operation Pods are rejected rather than silently broadening scheduling. ServiceAccount token projection is also rejected because every operation template sets automountServiceAccountToken=false. Arbitrary mutating-webhook changes remain outside the envelope. Priority, RuntimeClass, LimitRange, ServiceAccount image pull secrets, DefaultTolerationSeconds, ExtendedResourceToleration, and AlwaysPullImages are the explicitly modeled mutations. What a service mesh or a policy engine asks a Pod to carry is declared in spec.execution.podMetadata, and a Pod the API server refuses anyway is reported with reason PodAdmissionRefused; Execution has that contract.

PodLevelResources is enabled by default in the supported Kubernetes window, but those releases apply LimitRange default and defaultRequest only to Container items. Operation templates leave spec.resources unset, and the controller rejects a non-nil Pod-level resource stanza before dispatch. A future or customized API server that injects Pod-level resources fails closed until that mutation has an explicit, versioned snapshot model.

Admission does not trust an objectSelector, because a Pod creator controls those labels. Its match condition selects either side of an update when the Pod has the complete managed identity or a controlling Job whose name has one of the reserved operation prefixes. A label-less clone of a real operation Pod therefore still reaches the webhook, while unrelated non-operation Job Pods remain outside its outage scope. The handler then requires an exact Job controller owner and binds that Job to the current PtahSchema, operation, and persisted Pod intent. The same scope covers the ephemeralcontainers and resize subresources.

On create, the webhook also requires the built-in Kubernetes Job-controller identity. The Pod must preserve the full <job-name>- generateName, while its concrete name must use the API server’s effective prefix (at most 58 characters) followed by exactly five lowercase alphanumeric characters. This prevents a namespace Pod creator from cloning an active Job’s owner UID, labels, and executable template. The controller independently rechecks the same generated-name chain before accepting terminal evidence or recovery state. Only the Job controller may remove its tracking finalizer, and that exception permits no other Pod mutation.

While the webhook is unavailable, managed Pods and Pods controlled by Jobs using a reserved operation prefix fail closed, but unrelated non-operation Job Pods remain available. The controller and certificate-rotation Deployments remain recoverable because their Pods are ReplicaSet-owned. Run at least two controller replicas and preserve the PodDisruptionBudget during maintenance.

The managed-identity and Job-owner match condition is evaluated after mutating admission. A cluster administrator who can install or change a cluster-scoped mutating webhook is therefore inside the admission trust boundary: such a webhook could rewrite both the managed identity and reserved Job owner before validation. Ordinary workload creators cannot use missing or copied labels or extra owner references to gain operation admission; matching requests enter the rule, and the handler rejects foreign or ambiguous ownership.

Webhook certificate lifecycle

For a chart-generated webhook Secret, the chart stores tls.crt, tls.key, ca.crt, and ca.key and schedules a separate certificate rotator. The manager volume projects only tls.crt and tls.key; the CA certificate and CA private key never enter manager Pods. The manager ClusterRole has no Secret access. The rotator uses its own ServiceAccount with get and update limited by resourceNames to the generated Secret and a separate precreated Opaque staging Secret, plus one precreated coordination Lease and the exact mutating and validating webhook configurations. Both managed Secrets carry their one purpose label plus the exact Helm ownership label and the two exact Helm release annotations. The staging Secret has no owner references, finalizers, or stringData; every read and write requires that exact shape, a live UID and resource version, and an unchanged UID after update. Its chart manifest omits data, so pending private material is never copied into Helm release state. RBAC cannot limit which fields an update changes, so the rotator’s grant on the two webhook configurations reaches every field of both, and the release namespace contract is what bounds it, as it bounds every identity that namespace holds. The rotator itself changes only the caBundle of an entry. It treats its configured production webhook names as required identity anchors, then rotates every additional entry targeting the exact production Service. URL and foreign-Service entries remain untouched. This does not make a predecessor restartable once the upgrade has stopped the runtime; retry the same candidate to finish the interrupted transition. By default, certificateRotation.recreateMissingSecret=false: the chart grants no Secret create and renders no Secret-creation admission policy or binding. A deleted Secret therefore makes the rotator fail clearly and remain unready until an administrator restores it through a controlled Helm or GitOps operation.

Set certificateRotation.recreateMissingSecret=true only when automatic deletion recovery is required. Secret create cannot be restricted by resourceNames, so this opt-in adds a namespace-wide RBAC verb plus a fail-closed ValidatingAdmissionPolicy and binding that limit the rotator ServiceAccount to one exact TLS Secret name, namespace, two labels, two release annotations, and four nonempty data fields. A recovered Secret therefore stays owned by the same Helm release instead of becoming an unmanaged replacement. The API server applies the policy to the rotator’s request, so the opt-in grants the rotator no access to the policy or the binding themselves. The policy is written once, in Go, and the chart template that ships it is generated from that definition; the certificates acceptance suite proves the refusal against a live API server by creating an unrelated Secret as the rotator’s ServiceAccount and reading the denial back.

The single-release opt-in has a bootstrap tradeoff: Helm cannot atomically establish the admission policy and grant RBAC. A namespace-wide create grant can therefore exist briefly before policy enforcement is established. The rotator creates the Secret only after finding it missing, and an install renders the Secret, but nothing constrains a compromised ServiceAccount acting outside the rotator during that interval. Leaving the default disabled removes both the broad verb and this ordering window. Webhook endpoint discovery uses a read-only EndpointSlice list grant in the release namespace. Kubernetes assigns slice names dynamically and RBAC cannot constrain a list by label selector, so the rotator validates the inventory against the exact release Service identity before opening direct connections.

Serving certificates rotate before certificateRotation.renewalThreshold. The CA rotates before its threshold, when it cannot safely issue a full-lived serving certificate, when ca.key is missing, malformed, or does not match ca.crt, or when ca.crt is missing, malformed, or unrelated to the serving leaf. A Helm upgrade refuses a generated Secret without ca.key rather than rendering one, and the running rotator writes a new CA with its key. Candidate CAs from managed webhook entries are filtered certificate-by-certificate. A candidate is retained only when it signs the exact persisted leaf for every Service DNS name and every stable endpoint proves it is serving that byte-exact leaf. Unrelated roots are never copied between entries. An expired certificate follows the same recovery path; the rotator talks to the Kubernetes API and probes webhook endpoints directly, so recovery never depends on a successful call through the expired production admission webhook.

A new CA is published before anything presents a certificate it issued, and the old one is withdrawn only after nothing does. CA replacement takes three steps, and the rotator records each in the staging Secret before it takes the next:

1. Expand. The rotator atomically persists one versioned pending record, which binds the source Secret UID and a canonical digest of its four managed fields to the new CA and key and the serving certificate and key issued under it. Missing-Secret recovery records the absence explicitly instead of inventing a source identity. The rotator then adds the new CA to every managed entry, keeping that entry’s own parseable certificates and the CA that authenticates the certificate being served, and records when every entry held both. A current signer may be shared between entries only after its signature and exact live-leaf identity are independently proved; unauthenticated entry-local trust is never copied to another entry.
2. Switch. No earlier than certificateRotation.caSwitchDelay after the recorded time, one atomic Secret update replaces the serving certificate, serving key, CA, and CA key. A lost response is accepted only after byte-exact readback. The delay defaults to the reconciliation interval, six hours, and is at least one minute. An API server picks up a webhook configuration change within seconds, so by the switch every one trusts the new CA; the delay stands in for a proof that it does. The switch never waits past the current certificate’s life: it comes no later than certificateRotation.probeTimeout before that certificate, or every CA that verifies it, expires, which leaves the probe window for the new certificate to reach the manager Pods first.
3. Retire. The rotator lists the exact production Service EndpointSlices, rejects empty, unready, terminating, malformed, or duplicate endpoint sets, and performs a TLS handshake with every ready Pod IP using the Service DNS name. Once every endpoint serves the exact new certificate, verified by the new CA, and a second endpoint snapshot is identical, every managed entry trusts the new CA alone and the staging record is cleared. Removing a root needs no wait: an API server still holding the wider bundle trusts the served certificate all the same.

The rotator comes back at the switch time rather than a full interval later, and a restart neither resets nor skips the delay, because the expansion time lives in the staging Secret. If an entry has lost the new CA when the rotator reads it again, the rotator adds it back and the delay starts over. If the recorded time lies ahead of the rotator’s clock, the delay starts over from the clock the rotator has, so a clock that moved back cannot hold the switch off without limit.

The delay is measured on the rotator’s own clock, at both ends. A clock that is uniformly fast or slow does not change it. A clock that jumps forward during the delay, or a rotator Pod that moves to a node whose clock is ahead of the one that recorded the expansion, shortens it by that difference. Kubernetes certificate validity already assumes node clocks agree to within seconds; keep them synchronized and the delay holds with the same margin.

The delay protects a serving certificate that API servers can still verify, so it applies to a planned CA renewal. When the current certificate or every CA that issued it has expired, or no managed entry holds such a CA, admission through it has already stopped, and the rotator switches in the same pass as the expansion. When the generated Secret is missing, or its tls.crt and tls.key cannot be read, running manager Pods keep serving the pair they last loaded, but a Pod that restarts cannot load one at all and the rotator cannot tell what is served; it switches in the same pass as well, since hours of that cost more than the seconds an API server may take to pick up the expansion. A certificate that looks not yet valid only means the rotator’s clock runs behind the one that issued it, and does not shorten the delay. Renewing the serving certificate under an unchanged CA needs no delay either: the new certificate needs exactly the trust the old one had.

The install’s bootstrap material does not wait either. The chart renders a two-day CA, which is inside the renewal threshold from the start, and the rotator replaces any CA issued inside the threshold in its first pass, before it reports ready. The chart refuses a certificateRotation.renewalThreshold under 48 hours for that reason. helm install --wait therefore returns with the transition finished. Otherwise the rotator’s last bundle write would land a delay later, and if a helm upgrade were running then, between rendering the webhook configurations and applying them after its pre-upgrade hooks, the upgrade would fail with a server-side apply conflict on caBundle. The same conflict remains possible when a planned renewal or a recovery writes the bundles during an upgrade; rerunning the upgrade renders the new bundles and succeeds.

If the rotator stops between steps, its replacement validates and reloads the same byte-exact pending material, classifies the primary Secret against the stored source UID and digest, and continues from the recorded step. A primary Secret that already holds the pending material means the switch happened, so the replacement proves the endpoints and retires the old CA. The rotator never generates a second candidate while a temporally usable pending record exists, even when the old certificate crosses its own renewal threshold during the delay. It validates durable cryptographic identity separately from certificate time, then classifies the exact primary relationship before acting. If related staged material has expired or is not yet valid, the rotator atomically clears only that record, confirms the clear by exact readback, and retries from the authoritative primary state; an unrelated record remains untouched and fail-closed. Ordinary rotation never configures the API server to trust neither the old nor the new serving certificate. controller-runtime watches the projected tls.crt and tls.key files and reloads them without restarting the manager Pods.

A generated Secret containing valid material with an empty or Opaque type is normalized to kubernetes.io/tls. The no-rotation path updates only the Secret type and the four managed material fields before repairing trust; rotation paths normalize the type as part of their existing atomic material update.

With missing-Secret recreation enabled, the rotator first appends a newly generated CA to each exact entry. It preserves every parseable certificate candidate from that entry even when neighboring bytes are malformed, without borrowing trust from another entry. A manager Pod that restarts while the Secret is gone cannot mount its certificate, so the rotator does not wait out the switch delay: in the same pass it recreates only the policy-constrained Secret, accepts an uncertain or racing create only after byte-exact read-back, proves the new leaf at every stable endpoint, and contracts every entry to the new CA. An API server that has not yet picked up the expanded bundle refuses the matching requests until it does, usually for seconds. Admission remains fail-closed while Pods reload the recreated certificate. Manager readiness is tied to the webhook server’s started checker rather than a process-only ping.

The conservative defaults reconcile immediately at startup and every six hours, bound each reconciliation to 15 minutes, and retry failures with jittered exponential backoff from five seconds to five minutes. Liveness reports whether the supervisor is running; readiness is true only after a successful complete reconciliation and becomes false during retries. The Deployment exposes those states on /healthz and /readyz without serving certificate or key material.

The binary accepts those timings only when the operation deadline strictly exceeds Lease acquisition plus two endpoint-probe windows, and when the interval, the operation deadline, and the CA switch delay together stay below the renewal threshold. That keeps a planned renewal noticed a whole interval late from ever needing its full delay past the threshold; the switch deadline above is what keeps it ahead of the certificate’s expiry. Invalid or overflowing timing combinations fail at process startup instead of creating a permanently unready rotator.

It renews 30 days before expiry, issues 90-day serving certificates and three-year CAs, and allows five minutes for Secret projection plus endpoint proof. probeTimeout must exceed probeInterval; serving validity must exceed the renewal threshold; CA validity must exceed serving validity. The separately authorized, ReplicaSet-owned Deployment is deliberately outside the all-Job admission rule, so it can restart and repair trust when the webhook certificate or CA bundle is already broken. Its precreated Lease serializes reconciliation. To request an immediate reconciliation without changing the interval, restart that Deployment. A restart does not bring a pending CA switch forward:

kubectl -n <namespace> rollout restart \
  deployment/<certificate-rotation-deployment>

Never copy ca.key into diagnostics. Inspect Deployment status and structured error messages; certificate and key bytes are intentionally absent from logs.

Normal status progression

Use the phase as a summary and Conditions as the stable machine interface:

kubectl -n <namespace> get ptahschema <name>
kubectl -n <namespace> get ptahschema <name> -o yaml
kubectl -n <namespace> get ptahschemaplan
kubectl events -n <namespace> --for ptahschema/<name>

The complete stable condition-reason vocabulary is cataloged in Condition reasons. Automation should compare the type, status, and reason tuple and require observedGeneration to match the resource generation; condition messages are diagnostic text, not an API.

status.target.driftFindings is a canonical summary of the most recent raw Observe result, one entry for each category the report found. Entries are ordered by descending severity and then category and contain no object names or SQL. A change to an object one of Ptah’s feature owners plans, such as a PostgreSQL extension, is counted under feature_changes followed by the type of object, as in feature_changes:ptah.run/pgextension/extension-change, and an owner’s verdict on whether the change widens access under feature_access_widened, feature_access_narrowed, feature_access_unchanged or feature_access_unknown with the same suffix. The type names a kind of object, never the object. Objects the comparison could not decide are counted under undecided, with severity warning.

The list is never cut short. It holds up to 128 entries, and every category without a type fits under that bound, so driftFindingCount is always the sum of the entries’ counts. A report whose types take it past 128 categories is refused, and Observe fails rather than publish part of it. A subsequent scoped Plan deliberately does not replace this raw observation summary: the DriftDetected condition describes the authoritative managed scope, while status.target remains evidence for the observation identified by driftReportDigest and lastObservedAt.

Drift does not always come with a finding. Should a report find drift it lists under no category, it is observed as drift with no driftFindings, a driftFindingCount of zero and a highestDriftSeverity of safe, which is what the report rates a list with nothing in it. highestDriftSeverity is set whenever the report found drift and absent when it found none, so read it, not the count, to tell the two apart. Planning runs either way, and the plan names the statements and the privilege kinds they change. Since v0.16.0 Ptah lists grants, views, triggers and sequences under categories of their own, and a default privilege under feature_access_*.

Findings cover what the login can read. On PostgreSQL that is every object, whoever created it. On MySQL a trigger or a routine somebody added by hand can be in neither the findings nor the inventory, so an empty list does not prove that nobody added one. What the login cannot see says what each account misses at the pinned executor, and what it takes to see it.

status.target.inventory counts each kind of object the same observation compared, on both sides: how many the desired schema declares and how many the comparison found in the database, within the compared scope. A kind says which findings it joins. Most kinds are the stem of a pair of categories, so tables counts what tables_added and tables_removed are found among. Objects a feature owner compares, such as PostgreSQL extensions and policies, are counted under the kind their qualified categories carry, so ptah.run/pgextension/extension-change counts what feature_changes:ptah.run/pgextension/extension-change and feature_access_*:ptah.run/pgextension/extension-change are found among. A kind with no finding counts the same on both sides; the columns of a table found only in the database are not counted. A kind neither side holds is left out. The counts name no object. The inventory is written and replaced with the findings, and a Ptah that does not count leaves it absent.

In this example the database holds one table too many and none of the one extension the schema declares:

status:
  target:
    driftFindings:
      - category: tables_removed
        count: 1
        severity: destructive
      - category: feature_changes:ptah.run/pgextension/extension-change
        count: 1
        severity: safe
    inventory:
      - kind: columns
        desired: 87
        observed: 87
      - kind: ptah.run/pgextension/extension-change
        desired: 1
        observed: 0
      - kind: tables
        desired: 11
        observed: 12

Declared reference rows are counted in the same list. A row the database is missing, one whose managed columns someone changed, and one the declaration no longer holds appear as data_rows_inserted, data_rows_updated and data_rows_deleted, each a count with a severity and nothing else: no key, no column name and no value. Which rows those are is in the plan, and reading a data plan is data access. kubectl ptah schema restates the three as its reference-data line. The counts say that rows drifted; the plan decides what to do about it, so an unconverged row keeps InSync off and produces a non-empty plan even when no DDL changed.

EngineSupported=False with reason UnsupportedEngine is an explicit non-authorizing state, not a reconciliation crash. It creates no operation Job, plan, or approval and does not read the database credential. The controller finishes any already-dispatched Apply and its mandatory read-only proof before entering that state, while an undispatched Apply claim loses authorization without creating a Job. Selecting a supported engine later starts again at Resolve rather than reusing the old plan or approval.

Ready=True and InSync=True mean a read-only observation matched the verified artifact. They are not inferred from an apply exit code. ReadyToApply means the Always policy accepted a plan that destroys nothing and changes no privilege without claiming that an approval is needed; AwaitingApproval means an exact approval is required, and under Always reason PrivilegeChanges says the plan’s privilege changes are why. Blocked distinguishes deliberate policy refusal from an execution failure. ReadyToApply, AwaitingApproval, and Blocked all retain a read-only refresh deadline so the cadence survives controller restarts. A resource Blocked with TargetIdentityChanged retains a proof to read instead, at its failure retry interval. Once that deadline is due, a fresh resolution and observation take priority over an automatic Apply and over an approval the controller has not yet recorded on the plan. An approval already recorded for the current plan takes priority over the due read instead, because the read would mark it stale even when the plan did not change. The Apply it authorizes runs, and the read-only proof after it reads the database; Ptah refuses to apply a plan the database moved away from in the meantime. After policy and, when required, the exact recorded approval pass final validation, the controller captures one timestamp and stores it as the Apply operation’s start time. An automatic Apply first checks that timestamp against the deadline. That durable Apply claim is the one-shot authorization boundary: dispatch, lock contention, and controller restarts continue the claimed operation instead of letting a later timer reinterpret it.

The plan contract also binds what decides a plan’s meaning when it runs: the controller state version, the Ptah version, the executor image digest, and the runner protocol. If any of them changes before a mutating Job is dispatched, the current plan is no longer executable. A recorded approval becomes stale with reason ExecutionBindingChanged. A claimed but undispatched Apply releases its old authorization and database lock, the plan is cleared, and the replacement manager runs Resolve, Verify, Observe, and Plan again. Approve only the replacement plan; do not recreate an approval against the old UID or fingerprint. A dispatched Apply is never recreated under the new binding; its outcome is handled conservatively and requires post-Apply observation.

The same proof is required when an Apply runner refuses a target Secret that changed after approval. The refusal describes one Pod attempt; it cannot account for every attempt Kubernetes might have started for that Job. The operator keeps status.pendingObservation bound to the originally approved database. Reading the substituted database cannot settle it. Restore the original connection in the Secret and let the read-only proof finish. To move the resource afterward, change spec.target.urlFrom to a Secret for the new database, inspect its new plan, and approve that plan.

What the old binding left behind is listed in a PtahSchema’s status.pendingBindingRetirement until it is tidied: the retired epoch, the plan whose approvals are being marked stale, and the Job the retired claim dispatched. A read-only Job is left to finish and an Apply Job to stop before its cleanup is scheduled, and post-Apply observation starts only after that. The record then clears. A further change to the binding waits for it, so a rollout rolled back while an old Apply still runs takes effect once that Apply has stopped.

The manager’s own image digest and source revision, and the runner image digest, are recorded and not bound. A plan records the manager that published it and a Job the manager that dispatched it; neither is in a plan fingerprint or an approval. A Job is collected five minutes after it is harvested, or 24 hours after a failed Apply that keeps its log, so the harvest copies its dispatcher into status, beside the publisher the plan names: status.pendingObservation.dispatchedBy and then status.applied.dispatchedBy for a schema, status.lastRun.dispatchedBy and status.unresolvedRun.dispatchedBy for a migration. A manager release that changes only these – a patch or a security fix – keeps the execution epoch. A pending approval stays valid and is applied, and a published plan stays current.

Work the previous manager already dispatched is adopted under one rule: the replacement has to build the same Job, apart from the recorded manager identity. A claim’s live Job, in either family, is compared with the Job the replacement builds for it, with the manager identity taken from the live Pod template and the template held to the admission snapshot the claim persisted before dispatch. A release that changed nothing else in the Job adopts it. A release that also changed the Job or its Pod template – an environment variable, an annotation, a security setting – cannot confirm it. A dispatched PtahSchema Apply is settled as outcome unknown and the database is observed before anything else runs. A dispatched PtahMigration Apply is recorded as an Unknown run in status.unresolvedRun, and no further Apply is claimed until a History reading or a PtahMigrationRunAcknowledgment accounts for it. A dispatched read-only Job of either family is run again under a new attempt. A PtahMigration claim that recorded the Job’s UID, with the Job the one it fixes, waits for that Job to stop first and gives it the five-minute TTL of a harvested Job, because admission admits the TTL only on the Job an active claim names. Progressing names the Job while the claim waits. The wait lasts at most the Job’s spec.execution.activeDeadlineSeconds, 15 minutes by default, from the Job’s start, plus the termination grace its Pods are given and 30 seconds. A Job admission would refuse, such as one under another execution epoch, is left at once, and so is one that has not finished by then, such as one whose Pod stays Terminating on a node that is gone. A MigrationJobCleanupDeferred Warning Event names the Job it leaves.

A claim that had not dispatched yet resolves its admission snapshot again from the template the replacement builds, and the AdmissionSnapshotRefreshed Event says so. That happens once per claim: a template that changes again retires the claim, because a builder that does not build the same Job twice would otherwise refresh it forever.

The runner is built from the operator’s source and ships in the manager’s release, so its digest changes with every release, and binding it would bind the manager by another name. What the runner enforces inside the Pod – the arguments and environment it accepts, the checks around the executor, and the result frame it returns – is versioned by the runner protocol instead. A release that changes any of it bumps the protocol, and that retires the plans and approvals made under the old one.

The audit-visible status.executionBinding records those four components plus an opaque epoch. The epoch changes whenever one of them changes, including a rollback to a byte-identical set. A plan carries the epoch as spec.executionBindingID and binds it into its fingerprint, and an approval names the plan by that fingerprint, so approval is one-shot for that exact transition: it cannot become valid again after a later rollout or rollback, even if all four components return to their previous values.

A normal chart upgrade has a hard revision boundary: the Recreate strategy terminates every old manager Pod before any replacement manager Pod starts. Before termination, the old manager may still dispatch an approval that is valid for the complete old execution binding. That Job remains internally consistent and is never changed to the replacement binding. If an upgrade maintenance window requires that no new Apply be dispatched after the window begins, first scale the manager Deployment to zero and wait until all manager Pods have terminated. Then run helm upgrade --wait; the chart restores the configured replica count. When the release changes the execution binding, the replacement manager invalidates every undispatched old-binding authorization before it can mutate a database. When it does not, those authorizations carry over, and the scale-down only holds dispatch for the length of the window.

Moving to a new artifact

A new revision of a schema or of a migration sequence reaches the cluster as a new artifact: publish it, then point the resource at it. kubectl ptah set-source is the second step:

REF=$(ptah schema push oci://registry.example/acme/storefront:1.5.0 \
  --schema-file schema.sql --dialect postgres --format reference)
kubectl ptah set-source schema/storefront -n application --oci "$REF" --wait

ptah schema push --format reference and ptah migrations push --format reference print the reference of the manifest the push wrote, pinned by its digest, and nothing else. ptah oci resolve <reference> prints the one a tag names now, which a later push may already have moved, so prefer the push’s own answer. A push before Ptah v0.17.0 has no --format and prints the digest in its receipt; ptah oci resolve on the tag just pushed pins it, provided nothing pushed to that tag since. set-source takes oci://HOST/REPOSITORY@sha256:<64 hex> and nothing looser: a tag, a tag beside a digest, something after the registry that is not a repository path (another reference pasted there, say), or anything the operator’s own parser refuses is refused before the cluster is read, with exit status 2. The refusal names which. It resolves no tag itself.

It sets spec.desired.ociRef of a PtahSchema or spec.artifact.ociRef of a PtahMigration and no other field. The verification policy, the registry credential, the transport and the apply policy stay what they were, and the new artifact is held to them: the report names them under Kept, beside the reference it replaced and what the operator had observed. When the field already holds the reference, it sends nothing, and a wait that follows counts only the next reading of the database, which kubectl ptah refresh asks for.

What follows is the operator’s ordinary cycle at the new generation: it resolves the reference, verifies the artifact against the policy, reads the database or the migration history, and plans. A plan or an approval of the previous artifact does not carry over. The apply policy decides the rest:

spec.policy.apply	The new plan
OnApproval	waits for an approval naming it
Always	runs without a person; a schema plan that is destructive or changes privileges still waits for an approval
Never	is recorded, and nothing applies it

Under Always, then, set-source can be all it takes for a new revision to reach the database. An artifact that asks for nothing the database lacks leads to no plan, and the resource returns to InSync.

A suspended resource starts no Job, so none of this happens until it is resumed; set-source stores the change and its report says so.

--wait stays until the operator has planned from the new reference: a plan computed from it waits for a decision, or the resource rests in sync from it, whichever comes first. --for approval or --for in-sync waits for that one alone, and a wait for approval stops with exit status 8 when the resource rests in sync instead, since no plan from the change will wait. A plan the resource’s own policy holds – a schema plan that is destructive while spec.policy.allowDestructive is false, or any plan under apply Never – stops either wait with exit status 8 once the operator has published it, with the reason and the operator’s message: no approval is asked for and nothing applies it until the spec changes. Every ending needs the operator to have observed the generation the change made and a reading that began after it, and a document that carries another artifact does not count. A verification policy that refuses the new artifact stops either wait with exit status 8 too, without a reading, since it refuses before the database is read. A wait whose resource is pointed somewhere else meanwhile stops with exit status 8, as any wait does on a target it can no longer reach.

set-source changes nothing, and exits 4, for:

• A ConfigMap or Git source. Moving to an artifact replaces that source and its settings and names the verification policy the artifact is held to, which is an edit of the resource.
• A resource with an Apply whose Job may already exist. The change would move the generation the Apply was decided from, and the operator records an Apply whose inputs changed while it ran as an outcome nobody established, without reading its result. For a migration that is a run nobody accounted for, which a reading clears only once nothing of the artifact is left to apply. Wait for the Apply to end, with kubectl ptah wait migration/orders --for applied -n application, and run set-source again. An Apply claimed with no Job yet is not refused: the operator drops it when it sees the change, and its plan does not run.
• A source another writer changed between set-source’s read and its patch. The patch carries the resourceVersion of that read, so the API server refuses it when anything wrote in between. set-source reads again and retries over a write that changed nothing it decided from, such as the operator’s own status, and reports another writer’s reference rather than replacing it. Another writer’s change to the reference asked for is the change made: it says so and exits 0. A resource that keeps changing under it for three tries ends with the API server’s conflict and exit status 5.

It needs get and patch on the resource, the verbs of whoever writes desired state, and it changes the spec: with approvals.requireDistinctApprover on, whoever runs it is recorded as the resource’s last spec writer and cannot approve the plan that follows (Refusing a self-approval).

For a change made once in a while, kubectl edit is the same change in an editor:

kubectl edit ptahschema storefront -n application

spec:
  desired:
    ociRef: oci://registry.example/acme/storefront@sha256:<digest>

A PtahMigration holds the same line under spec.artifact. A resource kept in a repository changes that line in its manifest, applied the way the rest of it is, and that edit is also how a source moves between a ConfigMap, Git and an artifact. A script without the plugin can patch the field, which checks none of what set-source checks:

kubectl patch ptahschema storefront -n application --type json \
  -p '[{"op":"replace","path":"/spec/desired/ociRef","value":"oci://registry.example/acme/storefront@sha256:<digest>"}]'

Mutable tags and registry outages

Every reconciliation interval resolves the requested reference again. A moved tag clears dependent plan and applied evidence, then repeats verification and observation against the new digest. An old approval cannot authorize the new plan. This cadence also applies while a plan is ready to apply, waiting for approval, or Blocked: read-only verification, observation, and planning continue, the same immutable plan returns as current when its inputs are unchanged, and no Apply Job is created during the refresh. A moved tag or database drift observed by a due refresh therefore invalidates stale plan or approval evidence before mutation can begin. An approval the controller already recorded for the current plan is an exception: the Apply it authorizes runs first, with the plan built from the digest resolved before, and the source is resolved again one interval after the proof that follows the Apply. A tag that moved in the meantime is seen then. A resource Blocked with TargetIdentityChanged is another: it resolves no source until its proof ends.

Native verification always inspects the resolved digest. When the selected policy requires require_digest_pin, the runner also evaluates the original requested reference: a tag or implicit latest is refused even though it resolved successfully, while an explicit digest remains eligible. Policies that permit tags retain the refresh behavior above without weakening any later digest binding.

Registry failures are fail-closed. During a failed refresh of a previously resolved source, the last source, target, plan, applied record, and successful reconciliation timestamp remain historical evidence; they are not a claim of currentness. ArtifactResolved=Unknown with reason RefreshFailed, while PlanReady and InSync become Unknown with reason SourceFreshnessUnknown. No Verify, Observe, Plan, or Apply follows that failed Resolve. After connectivity returns, the operator must resolve and verify again, then observe and plan. A same-digest, no-drift recovery ends with PlanReady=False/NoChanges, InSync=True/ScopedConverged, and no Apply. Observe and plan also need to fetch the digest-pinned schema, so an outage can delay database checks without causing mutation.

Every registry-authentication Secret must include registry: <host[:port]>, matching the OCI client’s effective request authority. This applies to both environment-key and Docker config JSON representations; the latter retains its standard .dockerconfigjson key alongside the fixed grant. A source under oci://docker.io/... therefore grants registry-1.docker.io. Missing, malformed, or mismatched authority grants stop the Job before Ptah can make a registry request. Authenticated plain HTTP also requires allowPlainHTTP: "true" in the authentication Secret. Rotate those values only when intentionally changing the credential’s authority or transport grant.

An authenticated source with transport.caFrom must also place caSHA256: sha256:<64 lowercase hex> in that same registry Secret. Compute the digest over the exact selected ConfigMap value, including its final newline if present. Missing or changed grant bytes fail before the registry client starts. The operator snapshots the selected CA once per Job, so a projected ConfigMap update cannot change trust roots between validation and fetch. Rotate a custom CA and its Secret-owned digest grant together; until both projections agree, read-only source operations fail closed. Anonymous custom-CA sources do not need a Secret grant but use the same per-Job snapshot boundary and 1 MiB limit.

Digest-pinned references reduce change ambiguity and are recommended for production. Promote the same digest between environments rather than rebuilding equivalent tags; kubectl ptah set-source takes nothing else.

External database targets

The operator does not provision a database. An operation Job only needs a network route to the target and a namespace-local Secret containing the URL selected by spec.target.urlFrom. That target may be a managed service, a private endpoint, or a database outside Kubernetes. Keep the Secret scoped to the schema namespace, make the selected key nonempty, and set one stable coordinationKey for every URL alias that reaches the same physical database.

A selectorless Service plus a manually managed EndpointSlice is one way to give an external address stable in-cluster DNS. Bind the EndpointSlice to the exact Service UID, set a distinct endpointslice.kubernetes.io/managed-by label for the component that owns it, publish only the database port, and arrange lifecycle management for address changes; the operator deliberately does not rewrite that route. NetworkPolicy, firewall rules, TLS, database privileges, backups, and high availability remain the platform owner’s responsibility. The runtime role needs enough DDL authority for the requested migrations but should not be a database superuser. Database ports do not need to be exposed on a Kubernetes node or on the host running a kind test cluster.

Suspension and deletion

Setting spec.suspend=true prevents new Jobs and cancels no SQL. If an apply is already active, the controller continues observing that Job until its outcome is safe to classify; it does not delete the Pod mid-mutation.

For either kind that classification is Unknown, whatever the run did. Suspending is a spec edit, the generation it bumps is one of the inputs the Apply claim was decided from, and a claim whose inputs changed while its Job ran is recorded without its result being read.

A PtahMigration records it as a run nobody accounted for. The Lease goes back once nothing the run dispatched can still write. The record stays while the resource is suspended, because a suspended migration takes no reading, and it clears on the first reading after resume that finds nothing of the artifact pending on the same database.

A PtahSchema records no applied plan for it, and owes a proof that reads the database, which waits for resume and for the Apply’s execution deadline and grace period. To have the run’s own result recorded in either kind, let it finish before suspending: kubectl ptah suspend names an Apply it finds running, and kubectl ptah wait migration/orders --for applied -n application waits for one to end and the operator’s reading afterwards.

While a PtahSchema owes the proof of an Apply, suspension starts no proof Job and drops a proof claim whose Job was not created; a proof Observe already running ends, and the Plan that completes the proof waits for resume. The schema retains and renews the Apply’s database-realm Lease meanwhile. This deliberately blocks later mutations in the same realm until the resource is resumed and the read-only proof completes, or until the resource is deleted. Releasing the Lease while retaining unresolved proof would let an intervening Apply contaminate the audit result.

kubectl ptah suspend sets the field and nothing else, and kubectl ptah resume clears it:

kubectl ptah suspend schema/storefront -n application
kubectl ptah resume schema/storefront -n application

Each prints the state it asked for, spec.suspend and the generation that holds it, beside what the operator had observed when the API server answered: the generation it last acted on and its phase, and an operation still in flight, and what the suspension does to it. A read under way when the suspension arrives, a migration’s read of its history after a run included, is dropped rather than waited for, since it changes nothing; its Job stops at its own deadline. Running one again sends nothing when the field already holds the value. With --wait, suspend stays until the operator reports the resource Suspended with nothing in flight, and resume until the resource is back in sync from a reading taken after the change, or, with --for approval or --for applied, until a plan waits for a decision or an Apply has ended. Waiting for the operator has the rules every wait follows. The commands need get and patch on the resource.

For an edit made once in a while, kubectl edit is the same change in an editor:

kubectl edit ptahschema storefront -n application

spec:
  suspend: true

A resource kept in a repository is suspended by the same line in its manifest, applied the way the rest of it is. A script can patch the field:

kubectl patch ptahschema storefront -n application --type merge -p '{"spec":{"suspend":true}}'

Suspending or resuming changes the spec, so it moves the generation like any other edit. With approvals.requireDistinctApprover on, it also records whoever made it as the resource’s last spec writer, and that identity cannot approve the plans that follow (Refusing a self-approval).

Deleting a PtahSchema never runs SQL. The transient finalizer exists only to observe an already active operation and release coordination safely. It waits on a running Job only while the operation holds the database lock: an Apply, a Plan, or the Observe that proves an Apply. A Resolve, a Verify or any other Observe holds nothing, so deletion drops its claim without waiting, including one whose Pod a policy refuses. Once no operation is active, deletion removes Kubernetes-owned plans and Jobs through normal garbage collection; database objects remain untouched.

Deleting a PtahMigration follows the same rule, and an Apply is where it is visible. The Job is owned by the resource, so releasing the finalizer under a running Apply hands a live executor to cascading deletion and stops it between statements. So kubectl delete ptahmigration blocks while the dispatched Job is unfinished or a Pod it owns has not stopped. Once nothing can write, a claim that dispatched something records the run Unknown, because nothing read what it did; a claim that only reserved a Job name records nothing, because nothing ran. Both hand the database Lease back and then release the resource. A read-only operation is not waited on: its Job reads and reports, and deletion discards the claim and goes.

Use the default propagation. kubectl delete --cascade=foreground asks Kubernetes to remove the resource’s dependents before the resource, and the Apply Job is one of them, so the garbage collector takes the Job and its Pod before the operator is reconciled at all. The wait above cannot prevent that: the finalizer holds the owner, and the dependent goes first. Deleting a PtahMigration with a running Apply that way stops the executor between statements and leaves the database in a state only a person can account for.

How long that block lasts is bounded for the Job and not for the Pod. spec.execution.activeDeadlineSeconds is what turns an executor that hangs into a terminal Job, so a run that is merely slow ends on its own deadline. A Pod on a node the API server cannot reach is a different case: it stays Running until the node object goes or the Pod is force-deleted, and the deletion waits with it for as long as that takes. Nothing the operator holds ends that wait. The database Lease is renewed for the whole of it, so no other resource takes that database while a Pod that may still be executing SQL is unaccounted for – which is the reason the wait is worth its cost. Recover the node or force-delete the Pod; removing the operator.ptah.run/migration-operation finalizer by hand is the last resort, and it is a statement that nothing is executing against that database any more.

Deleting a namespace

A namespace that ran a PtahSchema or a PtahMigration stays Terminating for about an hour after its last operation. kubectl delete namespace --wait=true blocks for all of that time, and a namespace of the same name cannot be created until the old one is gone.

Each operation leaves PtahResultRecord objects behind: its result, and the credential that delivered it. The admission webhook refuses to delete one until at least an hour after it was written, for every caller, the namespace controller and a cluster administrator included. The hour is longer than the five-minute backup lag plus the thirty-minute recovery bound the operator is qualified against, so a recovery that starts right after an operation still finds what that operation reported. Backup and recovery says what the records are for.

The rest of the namespace goes first. What remains says why it waits:

kubectl get namespace <namespace> \
  -o jsonpath='{.status.conditions[?(@.type=="NamespaceDeletionContentFailure")].message}'

Failed to delete all resource types, 1 remaining: admission webhook "vcontrollerwrite.operator.ptah.run" denied the request: result publication retention has not authorized deletion: record ptah-result-17b56fc4280e1d0fb0c22f0e053a105826c6cc7423103c740b1b53302693d95d cannot be deleted before 2026-11-06T16:24:52Z

The time is the earliest that one record can go, in UTC. It is not when the namespace will be gone. The API server stops at the first record it is refused, so the message names the first by name, and the others can have later times. At the named time the operator checks again, and the next refusal can name a later one. The list says how long the whole wait is:

kubectl get ptahresultrecords -n <namespace> --sort-by=.metadata.creationTimestamp \
  -o custom-columns=NAME:.metadata.name,TYPE:.spec.type,CREATED:.metadata.creationTimestamp,DELETING:.metadata.deletionTimestamp

The last record can go an hour after the newest time in either column. DELETING is set where nothing had recorded the attempt as finished when the namespace’s deletion began, or where a full quota refused that record earlier. A terminating namespace refuses the retired record that would start the window, so the manager starts it with a foreground deletion of the record instead, and the hour runs from that time. A namespace deleted in the middle of an operation therefore waits about an hour from its deletion, however old the rest of its records are.

The manager’s collector removes each record once its window has ended, and the namespace follows.

A message that ends at has not authorized deletion, with no time, is not about time. The record’s operation may still be pinned by its resource, its original Job may still exist, the records it depends on may still have to go first, or the manager may have failed to read one of those. No time would lift any of them, so the message names none. In a terminating namespace they normally clear on their own, as the resource and its Job go and the collector removes records in order. If one outlasts the last window, the manager is the place to look: the collector runs on the leader. A failed calling webhook message means the manager is not answering at all; see Webhook certificate lifecycle.

What does not shorten it

With the operator installed, nothing does. These look as though they should:

• kubectl delete ptahresultrecord is refused the same way. With --cascade=foreground it is refused where a retired record exists, and where none exists it starts a window instead of ending one.
• Scaling the manager to zero leaves the webhook unanswered. Its failurePolicy is Fail, so every deletion is refused until the manager returns, and nothing collects the records meanwhile.
• resultDelivery.enabled=false changes nothing about the window. Retention runs the same with delivery off.

Do not remove the webhook configuration by hand to get past this. That leaves a release running without any of the guards the configuration holds, and it says nothing to whoever owns the installation.

While result delivery is off

A record outlives the setting that wrote it, so from 0.2.10 retention runs with resultDelivery.enabled=false as it does with delivery on. The collector runs on the leader, the manager holds the same grant on records, and the webhook judges every deletion by the same window, a credential record included. Operations write no records while delivery is off, so what this collects is what an earlier setting left. Nothing has to be turned on: a namespace holding such records reads the refusal above, with its time, and goes about an hour after they were retired.

A manager older than 0.2.10 runs no collector with delivery off, and its webhook refuses to delete any record but a credential one, with the message that names no time. A namespace holding one stays Terminating while delivery stays off. On such a release, turn delivery on until kubectl get ptahresultrecords -A lists nothing you are waiting for:

helm upgrade <release> <chart> --values <values> --set resultDelivery.enabled=true

That release’s chart refuses resultDelivery.enabled=true beside certificateRotation.enabled=false or a webhook.existingSecret, so such an installation has the uninstall below until it runs 0.2.10.

On kind with Kubernetes 1.37.0, a resource deleted with delivery off got its retired record on the collector’s next pass. A namespace deleted with delivery off read the refusal with a time within a minute, and the manager started its window with a foreground deletion. A release moved to webhook.existingSecret with rotation off did the same for a namespace deleted after the move. Each was over within a minute past the hour from that retirement or that DELETING time.

An installation that manages its own webhook trust, with webhook.existingSecret, runs the same collector. The chart refuses resultDelivery.enabled=true there, so this is the only way its records go short of an uninstall.

Tearing down CI and labs

Delete the cluster, when the cluster is disposable. Retention lives in the cluster, so it goes with it. kind delete cluster took two seconds on a single-node cluster, and this repository’s own harness does the same rather than wait.

When the cluster stays and the operator can go, uninstall the release first:

kubectl delete ptahschemas,ptahmigrations --all -n <namespace> --wait=true
helm uninstall <release> -n <release-namespace> --wait --timeout 5m
kubectl delete namespace <namespace> --wait=true

Nothing refuses the deletion once the webhook is gone. On kind with Kubernetes 1.37.0 the uninstall took 7 seconds, and the namespace, with 10 records inside their window, was gone 8 seconds later. Namespaces already Terminating followed within 40 seconds of the uninstall, including one refused for 30 minutes. Delete the resources first: a resource whose operation is active carries a finalizer only the manager removes, and with the manager gone the namespace would wait for that instead.

The uninstall takes retention away from every namespace in the cluster until the next install, which is why this is a path for a lab and not for an installation you keep. Records in other namespaces lose their protection at once, and those whose resource is gone can be removed by garbage collection before the next install.

A quota on result records

A ResourceQuota on count/ptahresultrecords.operator.ptah.run counts every record an operation leaves: the credential that delivered its result, the result, which is one record up to 256 KiB and more above that, and the retired record the manager writes once nothing pins the result. The records stay while the operation’s Job does and at least an hour after retirement, so the quota has to hold the records of every operation in that span.

Leave room in the quota for one retired record per operation. The manager writes it after the operation’s own records, so a quota sized to exactly what operations publish is full at the moment each one has to be retired.

When the quota refuses a retired record, the manager starts the window the way it does in a terminating namespace: it deletes the operation’s result and credential records in the foreground, and the hour runs from their DELETING time. It records ResultRetirementQuotaExceeded Warning Events on the PtahSchema or PtahMigration the operation belongs to, one for the result and one for the credential, and each names the quota and quotes the usage the API server reported:

kubectl events -n <namespace> --for ptahschema/<name> --types=Warning
kubectl describe resourcequota -n <namespace> <quota>

Once the hour has passed, the collector removes the result’s other records, and garbage collection removes the two deleting ones. Nothing is lost, and nothing goes sooner than it would have.

While the operation’s Job still exists, a foreground deletion cannot start the window either, because the Job could still deliver. The manager tries again after 30 seconds, doubling up to every 5 minutes, and each Warning says it retries. Free room in the quota, or delete the finished Job once you have read its log, and the next attempt goes through.

A quota with no room refuses an operation’s own records too: its credential, with a ResultCredentialFailed Warning Event, and its result, which then cannot be stored. A Job finished but its result is unavailable says what follows.

Failure recovery

• Read-only Resolve, Verify, Observe, and Plan failures retry after the bounded failure interval with a new deterministic Job attempt.
• A changed generation, policy, artifact, coordination key, route identity, or observed state discards the stale operation or plan before apply.
• Once a mutating child may have been dispatched, missing output, timeout, cancellation, or malformed output is OutcomeUnknown. The controller preserves the immutable Apply holder and permits only a fresh observation. If Job creation or identity is uncertain, observation waits for the complete possible mutation deadline while the holder is renewed. The plan is never blindly replayed.
• A migration Apply whose Pod was stopped – a drain, a preemption, an eviction, its deadline – is recorded with the account Ptah wrote as it stopped, when it managed to write one inside the Pod’s grace: a run stopped between two files is a failed run with the versions it applied, and the history read that follows confirms it. One stopped inside a statement that commits on its own, outside a transaction or a DDL statement on MySQL, is unknown, because Ptah cannot tell whether the server applied it. Where the log holding that account is gone, the summary in the Pod’s termination message stands in for it (The termination summary).
• A stale-plan refusal is read from the refusal Ptah’s apply report carries, whether the schema or the declared rows moved, and reported as stale_plan. It is still OutcomeUnknown, because another Pod of the same Job may have run: the plan is cleared, its recorded approval becomes stale, and reconciliation observes the database again.
• A controller restart resumes the persisted operation and existing Job UID. Replacement or unrelated Jobs are rejected.
• A successful apply transitions to VerifyingConvergence; only a later same-target observation can record status.applied and InSync=True.

The operator does not perform automatic rollback. Repair the desired artifact or database deliberately, then let observation produce a new plan.

Observation reads the database at the resource’s next interval. To read it as soon as no operation is running instead, after a repair, ask for a read:

kubectl ptah refresh schema/storefront -n application --wait

refresh writes the time of the request into the operator.ptah.run/read-requested-at annotation, later than any value the resource already holds, so two requests a moment apart are two requests. The operator takes it once no operation is running, the resource is not suspended and no retry delay is pending. A read-only operation that failed gives way to the request when its retry delay ends. A read that was already under way when the request arrived does not answer it, because it may have read the database before the repair. Without --wait the command says only that the request was written, which is not the read; with it, the command stays until the read the request asked for has ended. A script can write the annotation itself:

kubectl annotate ptahschema storefront \
  operator.ptah.run/read-requested-at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" --overwrite

status.lastReadRequest shows the request once it is answered. It never starts an Apply and never shortens a retry delay; Reading again on request has the rules.

Waiting for the operator

A change takes effect when the operator acts on it: it reads, plans and applies on its own schedule, and a condition can stay True long after the state it described has moved. kubectl ptah wait reads a resource until it reaches a target, and gives up at a timeout:

kubectl ptah wait schema/storefront --for in-sync --timeout 10m -n application

--for	Ends when
in-sync	the resource rests in sync at the generation it had when the wait began, from a reading of the database that began after the wait did
approval	a plan waits for an approval, with none recorded on it yet, at the generation the resource had when the wait began; a plan already waiting counts
applied	an Apply or a migration run that ended after the wait began is recorded applied, and the operator’s reading afterwards confirms it, for a migration the reading of the run’s database it records on the run as status.lastRun.verification; one that ended before does not count, even when that reading ends after; --plan narrows it to one plan, by name or fingerprint, and another plan’s run ends the wait
suspended	the operator reports the resource Suspended for that generation, with nothing in flight; a schema still owing the proof of an Apply reaches it, since that proof starts no Job until resume
read	the read a kubectl ptah refresh asked for has ended; --read-request names the request

Apart from a plan already waiting for an approval, nothing that was already true when the wait began counts. A schema that was InSync before somebody changed its database by hand stays InSync until it reads the database again, so waiting for in-sync after such a change waits for that reading: ask for it with kubectl ptah refresh --wait, or start the wait before the change. That reading covers what the login can see, so on MySQL a schema can stay InSync after a change the account cannot read. Approving a plan and refreshing change no generation, which is why a wait does not stop at the generation alone. A change made with the plugin waits in the same command with --wait, from the answer to the change itself; kubectl ptah approve --wait waits for the Apply of the plan it approved. For a migration it also waits for that approval: status.lastRun and the journal name the plan and the approval each run carried out, and the wait reads those rather than the order or the time of what happened.

It ends with	When
OtherPlanApplied	another plan’s run ended
OtherApprovalConsumed	the same plan ran under another approval, as when two people approved it and the operator took the oldest, or a run spent that other approval, which leaves this one Stale
PlanSuperseded	a migration’s plan will not run under this approval: another plan replaced it or is running, a reading left nothing to apply, or the operator marked the approval Stale, as it does when a read ends anywhere but its plan or the verification policy refuses the artifact; while that refusal holds, the wait ends with PolicyRefused instead
CompletionNotProved	a run ended that may be this one and no record says so: its record names no plan because an operator older than 0.2.10 wrote it, it names this plan and not what allowed the run, as the entry of a hold a restore brought back does, it names no fingerprint and --plan gave one, or a later run replaced it and the journal holds no reading after it. Also this very run, when the database moved before the next reading, since a reading of another database cannot verify it: the run has no verification and status.history.targetIdentityDigest names another database than the run’s. Or when a verification policy refused the artifact before any reading followed the run, since nothing reads the database until an artifact passes

--for approval asks whether a plan waits for you, so a plan that waits already answers it, even one computed before the wait began. Started right after an edit the operator has not read yet, it can answer with the plan from before that edit: an edit that moves the generation holds the wait until the operator has observed it, and one that does not, such as a ConfigMap’s, does not hold it. kubectl ptah approve shows the current plan, and refuses a decision about a plan that has been replaced since. set-source and resume wait from the answer to their own change, so a plan from before it does not count there.

A wait that times out exits 7 and prints the phase and the condition holding the resource as the last read found them, and what the wait still needed:

kubectl-ptah: ptahschema/storefront in namespace application did not reach in-sync within 10m0s.
  Phase:       AwaitingApproval
  Held by:     Ready=False, AwaitingApproval: Plan is waiting for approval
  Waiting for: phase InSync with InSync True at generation 4
kubectl ptah schema storefront -n application shows every condition with its message.

When the last reads failed, the timeout names that failure as well. The read a wait makes at its deadline is cut short by the deadline itself, and is not one of them.

A wait for in-sync started again after a timeout does not count a read that was already under way, because that read may have begun before the change being waited on. When the timeout found one, it says so and names kubectl ptah refresh schema/storefront --wait -n application, which asks for a read that begins later and waits for it.

A target the resource cannot reach any more stops the wait at once with exit status 8: an Apply that failed, a run whose outcome nobody established, a reading afterwards that did not confirm the Apply, another plan applied than the one named, a resource deleted or being deleted, or a wait for suspension on a resource whose spec does not ask for it.

A plan the resource’s own policy holds stops a wait for in-sync, approval or applied the same way: a schema plan that is destructive while spec.policy.allowDestructive is false, or any plan of either kind under apply Never. No approval is asked for and nothing applies it until the spec changes. The wait reads the refusal from the condition the operator keeps while it resolves, verifies and reads again at its interval – a schema’s ApprovalRequired, a migration’s Ready – so a wait that reads it mid-cycle, outside Blocked or Planning, stops as well. As with every target, the refusal counts only when it is for the generation the resource has now and was planned from a reading that began after the wait did:

kubectl-ptah: ptahschema/storefront in namespace application will not reach in-sync: PolicyBlocked.
  Phase:       Blocked
  Why:         spec.policy holds plan ptah-plan-0132bf7579a7c04e68e35110 (DestructiveChangesDisabled): Plan contains destructive changes and policy disallows them; read them with kubectl ptah plan, then either narrow the desired schema or set spec.policy.allowDestructive
kubectl ptah schema storefront -n application shows where it stands.

A verification policy that refuses the artifact stops a wait for in-sync, approval or applied with the reason PolicyRefused and the requirements it named. No plan outlives the refusal and no approval does, and the database is not read. A wait for applied whose migration run has already ended goes by that run instead: one that applied stops with CompletionNotProved, naming the run, since the database changed and the refusal holds back the reading that would confirm it. The wait reads the refusal from ArtifactVerified, False with PolicyRefused, for the generation the resource has now, at rest in Blocked, on two reads in a row. No reading dates it, so a refusal that stands when the wait begins stops it too. An older one does not: a spec edited since, a read request the operator has not taken, a read under way, or an artifact that passed since leaves the wait running, whatever Ready or Blocked still says.

To see whether a policy edit helps, ask for a read with kubectl ptah refresh --wait. The refusal that stands when you ask does not stop it. If the read it asked for ends in a refusal again, the wait stops with PolicyRefused and the requirements, as kubectl ptah wait --for read does. If it ends with a reading, the wait is done, even when a read on a newer request has been refused by the time the wait looks again. A refusal from before the request, or from a read that was already under way when it was written, leaves the wait running until the read the request starts ends; if that read does not end before the timeout, the timeout names the refusal if the resource still shows it.

The wait tells the two apart by a stamp the operator writes with each refusal. For a schema it is status.source.resolvedAt, which the Resolve of each read writes; for a migration it is the lastTransitionTime of ArtifactVerified, which the refusal writes. The wait compares that stamp with the one the resource carried when the wait began, or when the operator took the request if a read was under way at first. It never compares it with the time of the request, which your clock wrote and not the operator’s.

A wait does not put two of the operator’s times in order either, except on a run an older operator recorded. The manager that holds the lease writes them, and after a failover the next manager’s clock can be behind or ahead of the one before, so a later reading can carry an earlier time. A wait counts a reading by its stamp, status.target.lastObservedAt for a schema and status.history.observedAt for a migration: a stamp other than the one the resource carried when the wait began, or when the operator took the request, is a reading recorded since, whatever its time. Two readings stamped with the same second, which another manager’s clock can do to readings minutes apart, look like one, which can only make a wait longer.

A migration run is decided by its own record. The reading that confirms it records itself on the run, as status.lastRun.verification, in the same status write, so wait --for applied, --plan and approve --wait end at that record, in Planning when the reading found more migrations pending. Until a reading records itself there, a history of the run’s database is the reading from before the run, and a resource that rests Suspended or Blocked by its realm keeps the wait running. A history of another database, once the resource rests, is a reading that cannot verify the run, and the wait stops with CompletionNotProved. A run that an operator older than 0.2.10 recorded carries no operationID and gets no verification, so wait --for applied still dates its reading by the run’s finishedAt.

A resource its database realm blocks keeps a wait running: another resource’s change or an administrator’s grant ends that refusal, not a change to this one, and a timeout names it.

With -o yaml or -o json the command prints the resource as the wait last read it, and nothing when it did not reach the target. Waiting needs get on the resource and nothing else. kubectl ptah approve --wait on a migration also reads the approval it stored, which the approver ClusterRole lets it do; a read of the approval that is refused leaves the wait to the resource alone.

A migration run nobody accounted for

A PtahMigration whose Apply ended Partial or Unknown records the run in status.unresolvedRun and stops. The record is what a person needs in order to go and look, and it is deliberately somewhere a condition cannot reach: any later refusal rewrites the reason on Blocked, and a rewritten reason says nothing about whether the mutation was ever accounted for.

kubectl ptah migration orders -n application
kubectl get ptahmigration orders -o jsonpath='{.status.unresolvedRun}' | jq

The first prints the record under Unresolved run, apart from the last run, with the migration’s UID; the second is the record as stored.

The record names the attempt (operationID), the plan it was carrying out, and the credential-free identity of a database. It names the Job by name and UID wherever the manager established that one existed, and the Job and its logs may be gone by then, which is why the record carries their identity rather than pointing at them. A run that reported Partial or Unknown, or nothing, keeps its Pod log for 24 hours after the Job finishes, under apply: Always as under an approval. A claim that never recorded the Job’s UID, as when the manager stopped between creating the Job and recording it, records it before the TTL is asked for, so its Job keeps the log the same way, and a run recorded before the claim could read the Job’s name names the Job then. A Job under the reserved name that the claim does not fix, such as one under another execution epoch, is never recorded, and stays with no TTL. status.lastRun.message says how to read it when the Job had finished by the time the run was recorded. A Job still running then takes its TTL once it stops, and the message names no log.

Which database targetIdentityDigest names depends on what the run managed to say. A readable result frame reports the database the Pod’s URL named, and so does the summary the runner wrote into its Pod’s termination message, where it was read in place of a log that was gone. Without either – an Apply whose create was never confirmed, a Pod that wrote nothing a reader could use – the record falls back to the database the plan was computed against, which is the last one this resource read. The runner refuses an Apply whose URL names another target than its plan’s before Ptah starts, so a run whose frame names another one changed nothing, and the planned database is the only one a run can have changed. A Secret that rotated only its password still names the same database. The identity is the route the URL describes: the login, the host, the port, the database and the parameters that are not a credential or a connection setting, such as a timeout or MySQL’s checkConnLiveness and maxAllowedPacket. A host name that now resolves to another server is the same identity. What stops a run there is only the approved sequence the Apply carries: Ptah compares it with what it selects under the migration lock, so a server whose pending migrations are the approved ones runs them. Read that database before repairing anything: a reader who inspects another, such as a replica or a restored copy, finds nothing wrong and clears a record that was telling the truth.

jobName and jobUID are empty on one of those records as well: a create whose outcome the API server never confirmed. The name that claim reserved is not evidence that anything ran under it, so the record says nothing rather than sending a reader after a Job that may never have existed. The attempt and the plan are still there. The deadline that claim carried is not: it lived on status.activeOperation, which is cleared with the claim, and neither the record nor the plan copies it. What is observable instead is the Lease, which this resource keeps for as long as a dispatched Apply could still be writing.

While the record stands this resource publishes no plan and dispatches no Apply, including the migration that run was applying. It goes on reading unless something else has stopped it first: resolve, verify and the history read continue at the resource’s interval, and that history read is how the record clears. Four gates are answered before any operation is claimed, and a resource held at one of them never reaches the reading that would clear its record: suspension, an engine this operator does not support, a database realm another resource claims, and stored state written by a newer manager than the one running. The first three do not stop a person’s acknowledgment, which reads no database and is taken ahead of them.

An artifact the verification policy refuses holds the resource further along: resolve and verify run at the interval, and the history read does not start. kubectl ptah migration names it with the gates, as Blocked with PolicyRefused. The record stays, an acknowledgment still settles it, and the reading clears it once the artifact passes.

Passing those gates is not the same as getting the reading. Resolve, verify and the history read are retried for as long as they keep failing, with no attempt at which the operator gives up, so a resource failing one of them never reaches the history read either. If the record is not clearing after the database was repaired, read the Blocked condition first: a refused resource rests with no claim between its reads, so PolicyRefused is the answer and status.activeOperation is empty. Otherwise read status.activeOperation: the operation it names and the attempt it is on say whether the chain is stuck rather than the record, and an attempt climbing quickly is a Job failing as fast as it can be recreated. An attempt that restarts from 1 is a read that gave way and started the chain again.

Other resources may also be working against the same database if every claimant declares a shared realm. So the record stops this resource from changing the database; it is not a promise that the database is idle, and a repair should not assume one.

The run that caused the record is part of that. Retiring the claim does not stop a Pod: where the Apply may still be executing, this resource keeps the realm Lease until its dispatch and execution deadlines have passed, so the Lease in the coordination namespace is what says whether the run that is being accounted for could still be writing while it is accounted for.

How it clears

On its own, from a read-only reading of that same database with nothing of the artifact left to apply. That happens after a person repairs whatever the run left half-done – the recovery for a partial migration is to run what is left, or to undo what it committed, drop the unfinished revision, and publish the sequence without it – and the operator settles on its next reading with no spec edit.

A reading of a different database does not clear a record that names one, and a record always names one: the plan that run was carrying out was computed from a reading, and a reading that named no database is refused before it is stored. Neither does an artifact that ends before the database does: an artifact pointed at a shorter sequence says nothing about a run that went past it.

The other way it clears is a person’s acknowledgment, below: the case where the database is accounted for and still has work pending, which no reading can settle. Either way the resource records how in status.resolvedRun, and who, when a person did.

Nobody clears the record by writing status. The chart refuses a write to the status subresource of every operator kind from anyone but the manager’s ServiceAccount, a cluster administrator included, because status is what the controller and its admission webhooks decide from; a write that removed the record would also have recorded nobody.

Settling it by hand

Before you start

You need create on ptahmigrationrunacknowledgments in the resource’s namespace. The approver ClusterRole the chart ships grants it, since the decision is of the same weight as an approval: it lets the next plan be made against that database. Nothing cluster-scoped is touched.

Establish what the run did. The record is the operator saying it cannot tell, so acknowledging it without answering that question hands the next Apply a database in a state nobody checked.

Run it

Copy the resource’s UID and the run’s operation ID the operator published. kubectl ptah migration orders -n application prints both in its Unresolved run block, and so does:

kubectl -n application get ptahmigration orders \
  -o jsonpath='{.metadata.uid}{"\n"}{.status.unresolvedRun.operationID}{"\n"}'

and name them in a PtahMigrationRunAcknowledgment:

apiVersion: operator.ptah.run/v1alpha1
kind: PtahMigrationRunAcknowledgment
metadata:
  name: orders-run-accounted-for
  namespace: application
spec:
  migrationRef:
    name: orders
    uid: <metadata.uid of the PtahMigration>
  operationID: <status.unresolvedRun.operationID>

The admission webhook stamps your authenticated identity on it and refuses one written for anybody else, or one that names a run the resource does not record right now. The controller takes it on its next pass, whether or not the resource is suspended, and reads the database again before it plans anything.

What proves it worked

status.unresolvedRun and the operator.ptah.run/unresolved-run annotation are gone, and status.resolvedRun names the run, the acknowledgment and you:

kubectl -n application get ptahmigration orders -o jsonpath='{.status.resolvedRun}' | jq
kubectl -n application get ptahmigrationrunacknowledgment orders-run-accounted-for

kubectl ptah migration orders -n application prints the same as a Settled run line, and lists the acknowledgment with its conditions.

The acknowledgment reads Consumed=True, and a Normal Event, UnresolvedRunAcknowledged, names who acknowledged which run. The resource then plans whatever the database still lacks: under apply: OnApproval that plan waits for an approval like any other, and under Always it runs.

Where to stop

Do not acknowledge a run to make a resource move again. An acknowledgment the controller answers Stale=True settled nothing: it named a run the resource is not waiting on, usually because a reading settled it first, and a second acknowledgment of one run is answered the same way once the first has settled it. Do not reach for kubectl delete either; Deleting the resource discards it says what that costs.

If it fails

An acknowledgment admission refused says why: the operation ID the resource does record, a resource recreated under the same name, or a resource that records nothing to acknowledge. A record is written only where an Apply ends Partial or Unknown, so one that is back names a later run. Compare its operationID and jobUID with the run you acknowledged, and account for that run the same way.

A restore that drops status

The manager keeps a copy of the record in the resource’s own operator.ptah.run/unresolved-run annotation, as JSON. The copy is written before status.unresolvedRun and removed after it, so a stored record always has its copy. A backup tool that recreates the resource with its metadata and without its status – Velero does by default – brings the copy back, and the manager puts status.unresolvedRun back from it before it reads anything else, with a Warning Event, UnresolvedRunRestored. The restored record names the same run and clears the same two ways. The restored resource has a new UID, so an acknowledgment written before the loss does not carry over to it.

Once the resource exists, only the manager’s ServiceAccount may add, change or remove the annotation; the chart refuses anyone else, as it refuses a status write. Creating a resource that carries it is admitted, because a restore is exactly that. A copy the manager cannot read holds the resource Blocked with ApplyOutcomeUnknown and runs nothing; delete and recreate the resource once the database is accounted for.

Two restores keep nothing. Specs reapplied from Git carry no annotation, and a backup taken while an Apply was still in flight predates the record: the record is written when the run’s outcome is read. What protects those is the history read every rebuilt resource takes before it plans, and the approval a new plan waits for; Recover operator state says what to establish first.

Deleting the resource discards it

kubectl delete ptahmigration removes the record with the object. The operator does not refuse the deletion. A resource being deleted takes no further readings, so nothing but a person could clear its record; a refusal would be one the operator could never lift, and a resource nobody can remove is worse than a record that ends in the event stream.

It does refuse to lose it quietly. The pass that removes the finalizer emits a Warning Event, UnresolvedRunDiscarded, naming the outcome, the Job the run ran as, the plan it was carrying out and the database it addressed, and logs the same. Those are what remain for whoever later finds a change in that database nobody can account for, so capture them before the Event’s retention window closes:

kubectl -n "$NAMESPACE" get events --field-selector reason=UnresolvedRunDiscarded \
  -o custom-columns=WHEN:.lastTimestamp,OBJECT:.involvedObject.name,MESSAGE:.message

Establishing what the run did before deleting the resource is the better order. The record names everything needed for that, and it is the last reading of it.

Observability

The chart exposes the controller-runtime Prometheus endpoint through the optional metrics Service (metrics.service.enabled=true by default). The operator metrics use only closed, bounded label sets; object names, artifact digests, database URLs, SQL, and error strings are never labels.

Both resource families report through the same metrics, and family says which: schema for PtahSchema and migration for PtahMigration. An alert written without that label covers both, which is what an alert on an uncertain apply wants; add family="migration" to ask about one.

• ptah_operator_reconciliations_total{family,result} counts successful and failed reconciliations.
• ptah_operator_drift_observations_total{engine,outcome} distinguishes detected drift from in-sync observations. It carries no family: a migration reads its own recorded history rather than observing the live database.
• ptah_operator_plans_total{family,engine,destructive} counts published immutable plans. A schema counts a plan on the status write that records its publication in status.journal, so a scheduled read that clears the plan the schema waits on and publishes it again counts nothing, and the same plan counts again only once an Apply ran it or another plan took its place. A migration keeps the plan it waits on in status through its reads, and counts a plan its status did not name before. destructive is the schema planner’s conservative classification; a migration plan reports unknown, because nothing examines hand-written migrations for data loss.
• ptah_operator_approvals_total{family,outcome} counts approval transitions. required is a plan that asked for a decision, accepted is a decision the controller consumed at the dispatch boundary, and stale is an approval the controller marked unusable, one count each, because the plan it named stopped being current, the verification policy refused the artifact, or a run spent another approval of the same migration plan. A policy of Always waives the requirement and counts nothing, except for a schema plan that changes privileges, which still asks and counts as required. A schema counts required with each ApprovalRequired Event: once when a plan that asks for an approval is published, and once each time a plan already published starts asking, after something withdrew its requirement or after an edit of spec.policy.apply, which Events lists. A migration counts it each time its ApprovalRequired condition becomes True.
• ptah_operator_applies_total{family,outcome} counts started, completed, uncertain, and stale Apply transitions. A migration run that ended Partial or Unknown is uncertain: neither may be retried, and both leave a record that only a reading with nothing left to re-run, or a person, clears.
• ptah_operator_operation_duration_seconds{family,operation,outcome} measures completed logical operations, including uncertain and stale outcomes. operation is the union of both families: resolve, verify, observe, plan, apply, and history, the last of which only a migration performs.
• ptah_operator_failures_total{family,stage,category} separates infrastructure, configuration, operation, policy-change, stale-input, and uncertain-outcome failures by bounded state-machine stage. The stages are the operations plus controller, which is the operator’s own failure rather than an operation’s.

Every counter is incremented on a transition, not from the status a reconciliation read. A resource waiting for a person is reconciled on its interval for as long as it waits, so a counter read from its status would climb with the reconciliation cadence and report decisions nobody made.

Production logs are structured. Controller-runtime request context supplies the managed object identity; lifecycle records add the closed operation, numeric attempt, and phase fields. Reconciliation completion records include requeue and requeueAfter. Executor stdout and stderr are treated as untrusted secret-bearing data: only a strictly validated Plan may transport the plan Ptah saved. Resolve and Verify emit bounded typed evidence, Observe emits only summaries, and Apply emits no native output but the one line its Pod log keeps of a failure. Controller-facing failures are typed, and each carries a sentence of the runner’s own apart from these, which carry Ptah’s words:

• A desired schema or migration directory Ptah cannot load, under desired_source_unreadable. The words come from a second run of Ptah that holds no database URL and no credential, so they cannot quote the database. They can quote a fragment of the file. A desired file Ptah could not read says what a person does with them.
• A desired schema Ptah read and refuses as a whole, under desired_schema_invalid, when that fault is the one the failed command stopped on. The words come from the same second run and name what the file declares.
• A plan that would change a protected table, under protected_table. The words are the sentence of the schema plan report, which names the fenced tables.
• A Git source a Resolve cannot fetch, under child_exit. The words are the last line of native stderr from the pull, which names the refusal.

Each is redacted, stripped of control characters, joined onto one line and cut to a bound, and Output handling says what bounds each one and who it reaches. A failed Apply also writes Ptah’s words to its Pod log, and to nothing a controller reads, as the next section says.

Kubernetes Events provide the object-scoped audit trail for operation claims, completion, policy refusal, approval changes, and failures. Alert on a sustained increase in ptah_operator_failures_total and on any increase in ptah_operator_applies_total{outcome="uncertain"}. Use status Conditions and Events to identify the affected object instead of adding unbounded identity labels to metrics.

What a failed Apply leaves in its Pod log

When a schema or a migration Apply fails, the runner writes one line to the log of its operation Pod, the first line of what Ptah reported:

ptah-runner: ptah reported: apply schema changes: failed to execute SQL statement: SQL execution failed: ERROR: could not create unique index "[in the statement]" (SQLSTATE 23505)

That line stays out of status, Events, the result frame and the termination summary. It is there because it is the reason a person needs, and it can carry a role, a host, a database name and values read from a row. MySQL quotes the duplicate value, Duplicate entry 'alice@example.com' for key 'orders.orders_one_per_customer', and PostgreSQL quotes the value a cast refused, invalid input syntax for type integer: "not-a-number".

The plan’s own text stays out of the line. A database often quotes the statement it refused, and Ptah ends its sentence with that statement, so the runner compares each quoted part of the line with it and replaces what they share with [in the statement], before it cuts the line. That covers a literal, a password in a literal and a name the statement declares, which is why the example above does not name its index. A value read from a table is not in the statement, so it stays. The runner also removes the secrets in the Pod’s environment, in any case, and replaces control characters with a space, so a row cannot rewrite a reader’s terminal. Nothing can recognize a row as a row.

Anything that ships Pod logs elsewhere carries the line too, so a log pipeline receives row values from a failed Apply. Whoever holds pods/log in the namespace can read it, which includes the diagnostic reader role. A target failure with a known kind writes no such line, because the operator’s own sentence already names what to check.

How long the log stays depends on whether anything paces the runs:

The Apply	Its Job keeps the log
A person approved it, and it failed or its outcome is unknown	24 hours after the Job finishes
A migration run under apply: Always that stops the resource: it reported Partial or Unknown, it reported nothing, or it reported Failed and left a dirty revision row	24 hours after the Job finishes
Any other Apply no approval names, under apply: Always, that failed or whose outcome is unknown	5 minutes
It succeeded	5 minutes

A retained Job holds a place in a namespace’s Job quota, so the 24 hours go where Jobs cannot pile up. An approval needs a person each time. A migration run that reported Partial or Unknown, or nothing, stops the resource at once: it dispatches no Apply until a History reading or an acknowledgment settles the run. A run that reported Failed at a statement leaves a dirty revision row, so the reading that follows blocks the resource with HistoryDirty, and nothing runs until a person clears the row. Each is when a person has to read the database, and each holds one Job.

Nothing stops a schema that way. An uncertain Apply under Always goes to a proof, the proof plans again, and Always applies the new plan unless it needs an approval. A failure that goes on holds one Job per cycle, about 1,400 to 2,900 Jobs a day per resource at a 30 to 60 second cycle, and each run writes a fresh log, so every schema Apply no approval names keeps five minutes.

A migration run that reported Failed and left no such row keeps five minutes too, because nothing stops it. One that stopped before it selected a migration, as on a database it could not open, runs again after failureRetryInterval. One Ptah refused because the history moved after the plan ran nothing, and the next reading plans again.

What the run reported decides, not why the resource stopped. A run that reported Failed or Applied and is blocked because the resource was edited during the run, or because it opened a database other than the planned one, keeps five minutes. So does a run that reported Failed when its log held no result frame, because the termination summary read in its place does not say whether the run left a row. Suspending a resource after a failure that repeats stops the next run and does not lengthen the last one’s log. Read the log in that time, or ship Pod logs somewhere that keeps them. A plan that changes privileges needs an approval under Always as well, and its Apply keeps the 24 hours.

The Applying condition of a schema, status.lastRun.message of a migration and the Warning Event say where to read the log and for how long, from the TTL the Job holds:

kubectl logs -n application job/ptah-apply-orders-5d06b0eaaa30addb -c ptah

They say so only when there is something to read. A target that could not be opened, a refused or stale plan and an init container that failed write no such line, so the status points at none. A run whose result nobody read, because its frame was lost or its Job ran more than one Pod, says Anything the run wrote is in the Pod log. An execution binding change during a running Apply retires the claim with no Job to name, so a RetiredApplyLogKept Warning Event says the same once the Job’s Pods have stopped. A migration Apply whose outcome went Unknown while its Job was still running, as when a release changed the executor or the Job under it or the database lock changed hands, is recorded before admission lets the Job take a TTL, so its status.lastRun.message names no log. The manager patches the Job’s TTL by the table above once the Job stops, just before the write that retires the claim, and kubectl logs reads the Job status.lastRun.jobName names for as long as that TTL keeps it. Admission admits the TTL only on the Job a claim names by UID, so a claim that never recorded the UID, as when the manager stopped between creating the Job and recording it, records it before the TTL is asked for, where the Job is the one the claim fixes, and status.lastRun.jobName names it from then on. A Job that is not, such as one under another execution epoch, is never recorded. Where admission refuses the TTL, or the claim does not record the Job, the Job stays until it is deleted, and a MigrationJobCleanupDeferred Warning Event names it when the claim goes. When admission refuses the 24 hours, as the guard of the previous release does after a Helm upgrade that failed once the manager rolled out, the Job keeps five minutes, and a FailedApplyLogShortened Warning Event and a line in the manager’s log say so.

The 24 hours is how long Kubernetes keeps the Job. A node that is replaced takes the Pod’s log with it sooner, and deleting the resource collects the Job with it. A retained Job counts toward a count/jobs.batch quota, and a full quota stops every operation in the namespace, because each one needs a Job. Once you have read a finished Job’s log, kubectl delete job -n <namespace> <job> frees the quota and takes the Pod with it. The run is already recorded by then, so the resource goes on as it would have. The result records of its attempt stay at least as long as the Job does.

An Apply Job Kubernetes marked Failed, because its deadline passed or its Pod was lost, stays Failed for those 24 hours. A run that delivered its result exits zero, so a statement the database refused usually leaves a Complete Job, and this does not apply to it. If you run the KubeJobFailed alert from kube-prometheus, which fires when kube_job_failed stays above zero for 15 minutes, it fires for such a Job and keeps firing until the Job is gone. Five minutes never reached that alert, and 24 hours does. Deleting the Job once you have read its log clears it.

What is still unaccounted for

The counters above are events. They say an Apply ended uncertain once, in a process that may since have restarted, and nothing about whether one is still standing now. Four gauges answer that, rebuilt from durable status on every scrape rather than kept in memory, which is what makes them survive a restart and survive an unrelated refusal rewriting the resource’s conditions.

Series	What it is
ptah_operator_unresolved_attempts{family}	Resources carrying a record of a mutation nobody accounted for
ptah_operator_unresolved_owed_seconds{family}	Seconds since the operator could first have settled the oldest such record. Absent where a family has none
ptah_operator_unresolved_view_synced{}	1 when the leader read the complete state for this scrape
ptah_operator_unresolved_view_read_failures_total{}	Scrapes that could not read that state

A schema Apply whose runner reported it could not open the target counts here until the database answers and the proof completes, because one Pod’s report does not speak for its Job. DatabaseReachable=False, with a reason for the kind, says the target cannot be opened now. It does not say that nothing ran: an Apply that sent statements and lost the connection reads the same once its proof finds the database closed. Only the completed proof settles the record. The ApplyOutcomeUnknown Event carries the sentence when the Apply’s own Pod reported it could not open the target, and the fixed text otherwise.

The leader reads both resource families and both plan kinds directly from the API server once per scrape, sharing those results across the unresolved, resource-state and plan-store gauges. All four lists share a three-second budget. A failed or timed-out list suppresses every state gauge for that scrape and increments the read-failure counter; a warm controller cache cannot conceal the loss of API access. Followers publish no state. Losing leadership cancels an in-progress read. These are separate lists, not an atomic cross-kind snapshot. Plan chunks and ConfigMap payloads are not listed.

Every alert on these has to require the view first. A manager that has just started, or one whose read failed, publishes no counts at all and reports ptah_operator_unresolved_view_synced 0. An absent series and a series reading zero are the same thing to most query languages, and here they mean opposite things: one is “nothing is unresolved”, the other is “nobody has looked yet”.

ptah_operator_unresolved_view_synced == 1
  and ptah_operator_unresolved_attempts > 0

The gauges are aggregates, so they say that something is unaccounted for and not which resource. That is deliberate – an object name in a label grows the series with the cluster – and this is the drill-down:

kubectl get ptahmigrations -A -o json |
  jq -r '.items[] | select(.status.unresolvedRun != null) |
    "\(.metadata.namespace)/\(.metadata.name)\t\(.status.unresolvedRun.outcome)\t\(.status.unresolvedRun.jobName)"'
kubectl get ptahschemas -A -o json |
  jq -r '.items[] | select(.status.pendingObservation.outcome == "OutcomeUnknown") |
    "\(.metadata.namespace)/\(.metadata.name)\t\(.status.pendingObservation.applyJobName)"'

A migration named there is a run nobody accounted for and waits for a person. A schema named there settles itself once its horizon passes and a reading of the database confirms convergence.

The chart’s rules

With the Prometheus Operator, monitoring.prometheusRule.enabled renders a PrometheusRule with three alerts over these gauges:

Alert	Fires when	Severity
PtahOperatorUnresolvedApply	Any resource of a family carries an unresolved record, from the first evaluation that sees it	critical
PtahOperatorUnresolvedViewNotSynced	No replica reports a synchronized view, or no replica reports at all, for viewUnsyncedFor	warning
PtahOperatorUnresolvedViewReadFailures	A scrape failed to read the state within the last viewUnsyncedFor	warning

The first has no threshold and no pending period. There is no count or age of unaccounted work that is fine to ignore, and the records are durable, so the alert does not flap while one is being closed. The other two exist because the first cannot fire while the view is unsynchronized or unreadable.

Only the elected leader starts the view; every other replica reports it unsynchronized and publishes no counts. The rules take the max across replicas, which reads the leader and never counts a record twice.

viewUnsyncedFor has no default, and the chart refuses to render the rules without it. How long a manager takes to synchronize after a restart or a leader change depends on the cluster it runs in, so measure it there: restart the manager a few times and move leadership once (kubectl -n <release-namespace> rollout restart deployment/<release>, and delete the leader Pod), then read how many seconds the view spent unsynchronized in the hour that covers them:

sum_over_time((max(ptah_operator_unresolved_view_synced) == bool 0)[1h:15s]) * 15

Set viewUnsyncedFor comfortably above the longest single stretch you saw. The same number bounds the read-failure window, so a failure is reported for that long after it happened.

The rules are tested with promtool test rules against the scenarios in hack/testdata/prometheusrule/: one unresolved migration beside a follower, a healthy fleet at zero, a view that stays unsynchronized past the window, a leader change shorter than it, a lost scrape target, and a failed read.

The state gauges

The same view publishes the fleet as it stands, under the same guard: nothing while ptah_operator_unresolved_view_synced reads 0.

Series	What it is
ptah_operator_resources{family,phase}	Resources by the phase their status reports; Unset before the first reconciliation
ptah_operator_overdue_resources{family}	Resources, not suspended, past their own status.nextReconciliationTime
ptah_operator_overdue_seconds{family}	How far past it the latest one is. Absent where none is overdue
ptah_operator_active_operations{family,operation}	Resources with an operation in flight, by type. A failed attempt waiting for its retry is not in flight
ptah_operator_active_operation_seconds{family,operation}	How long the oldest of each type has been eligible to run, excluding a declared retry delay
ptah_operator_pending_lock_releases{family}	Resources still owing the release of a realm Lease
ptah_operator_stored_plans{family}	Plans retained in the cluster. The operator prunes none; pruning stored plans is the procedure
ptah_operator_stored_plan_bytes{}	Bytes the retained schema plans hold in their chunks, from each plan’s spec.size. A migration plan stores no chunk
ptah_operator_webhook_certificate_expiry_timestamp_seconds{}	When the admission certificate this replica presents expires; every replica publishes it
ptah_operator_webhook_certificate_read_failures_total{}	Scrapes that could not read or parse that certificate, which publish no expiry

Being overdue for a moment is normal: nextReconciliationTime passes when a pass starts, and the status moves once its Jobs have run. What is not normal is a resource that stays late, which is why the alert is on how late rather than on whether.

Each of these alerts renders only when its threshold is set, and the chart sets none:

Value	Alert
overdueAfterSeconds	PtahOperatorResourceOverdue
operationStalledAfterSeconds	PtahOperatorOperationStalled
lockReleaseOwedFor	PtahOperatorLockReleaseOwed
certificateExpiresWithinSeconds	PtahOperatorWebhookCertificateExpiring
planStoreBytesAbove	PtahOperatorPlanStoreLarge
failures.window with failures.count	PtahOperatorOperationsFailing
admissionFailingFor	PtahOperatorAdmissionUnavailable, which reads the API server’s metrics

Once PtahOperatorPlanStoreLarge fires, a missing state reading keeps the existing incident open. A fresh reading at or below the configured budget clears it; loss of API access alone cannot establish that plans were pruned.

Read them off the installation they are for, over a period that includes a rollout and a busy day, and set each comfortably above what normal work produced:

max_over_time(max(ptah_operator_overdue_seconds)[7d:1m])
max_over_time(max(ptah_operator_active_operation_seconds)[7d:1m])
max_over_time(sum(increase(ptah_operator_failures_total[30m]))[7d:5m])

What reaches a receiver

The rules are tested against synthetic series with promtool, and the path a real alert takes is tested on a cluster. The PostgreSQL migrations suite ends with a phase that stands up Prometheus, Alertmanager and a webhook receiver as plain Deployments, loads the rules from the chart’s PrometheusRule as a rule file, scrapes every manager Pod behind the metrics Service, and asserts what the receiver logged:

Condition the phase creates	Alert at the receiver	Asserted
An Apply Job removed while its run was going, from the migration rows	PtahOperatorUnresolvedApply{family="migration"}	Fires; the count in its summary equals what the drill-down above lists; its runbook link resolves to this page
A schema whose Resolve Pod no node will schedule	PtahOperatorOperationStalled{family="schema",operation="Resolve"}	Fires no earlier than the threshold and within the detection target; clears within the slack once the Pod runs
Every manager Pod deleted with every node cordoned	PtahOperatorUnresolvedViewNotSynced	Fires within viewUnsyncedFor plus the slack; clears once the managers are back

With a five-second scrape and evaluation interval and a five-second Alertmanager group wait, the phase declares a detection target of the rule’s own threshold plus 45 seconds, and a delivery later than that fails it. Those intervals are the phase’s, not a recommendation: a deployment that scrapes every 30 seconds has a target larger by the same amount.

The phase does not drive a certificate close to expiry, an admission webhook the API server cannot reach, or a failed upgrade hook. Their rules are tested with promtool only. PtahOperatorAdmissionUnavailable reads the API server’s own apiserver_admission_webhook_rejection_count, and a Job that a hook left failed is Kubernetes object state; both come from the cluster’s Kubernetes monitoring, not from this operator’s endpoint.

The supported integration is a Prometheus that scrapes each manager Pod, as the chart’s ServiceMonitor configures one to, and loads the chart’s rules, either as a PrometheusRule through the Prometheus Operator or as the rule file that object’s spec is.

Queries for a dashboard

A panel per question, over the series above. The state gauges come from the leader, so each reads the max across replicas; the counters are per process, so each sums them. The operation label on a state gauge is the API’s type (Apply), and on a counter it is the lower-case stage (apply).

Question	Query
What state is the fleet in	max by (family, phase) (ptah_operator_resources)
What needs a person	max by (family) (ptah_operator_unresolved_attempts) and max by (family) (ptah_operator_resources{phase=~"Blocked|AwaitingApproval"})
What has stopped being looked at	max by (family) (ptah_operator_overdue_resources) and max by (family) (ptah_operator_overdue_seconds)
How long operations take	histogram_quantile(0.95, sum by (family, operation, le) (rate(ptah_operator_operation_duration_seconds_bucket[1h])))
What is in flight, and for how long	max by (family, operation) (ptah_operator_active_operation_seconds)
What is failing	sum by (family, stage, category) (increase(ptah_operator_failures_total[1h]))
Whether a realm is held up	max by (family) (ptah_operator_active_operation_seconds{operation="Apply"}) and max by (family) (ptah_operator_pending_lock_releases)
How much history is kept	max by (family) (ptah_operator_stored_plans) and max(ptah_operator_stored_plan_bytes)
What the manager costs	process_resident_memory_bytes, rate(process_cpu_seconds_total[5m]) and workqueue_depth for the manager Pods

A claimed Apply that is waiting for its realm’s Lease holds its operation claim while it waits, so contention for a realm reads as an Apply in flight for longer than its Job would take, and PtahOperatorOperationStalled covers a wait that does not end. A PtahMigration also says so itself: Progressing=True with reason WaitingForDatabaseLock, naming the Lease, and the Job when one may still be running. Either family asks for the Lease every five seconds, so it goes on within seconds of the Lease coming free. The Lease names its holder: kubectl -n <operator namespace> get leases -o wide. Two resources claiming one realm without spec.target.sharedRealm is a refusal instead, and reads as Blocked with reason RealmConflict; a resource a PtahRealm does not admit reads as Blocked with reason RealmNotAuthorized.

Finding a resource that has stopped converging

The counters above cannot answer this one. They are aggregates over every resource the operator manages, so a cluster where one schema stopped reading two days ago and forty others are converging normally shows a healthy rate and nothing else. There is no per-object series to query, on purpose: an object label is unbounded, and a metric that carries one grows with the cluster.

What each resource does carry is its own answer, in its status:

kubectl get ptahschemas,ptahmigrations -A -o custom-columns=\
KIND:.kind,NS:.metadata.namespace,NAME:.metadata.name,\
PHASE:.status.phase,READY:'.status.conditions[?(@.type=="Ready")].status',\
NEXT:.status.nextReconciliationTime

status.nextReconciliationTime is when the controller intends to look again. A value in the past by more than one operation deadline is a resource nothing is working on, and the phase and conditions on the same row say why. Run it from a cron job, a kubectl plugin, or a script beside your alerting, and page on rows rather than on rates.

Turning that into a Prometheus series needs an exporter that reads object status and emits per-object gauges; kube-state-metrics does exactly this for custom resources, and its cardinality is then your choice rather than the operator’s. The reason of DatabaseReachable is what such a series can key on to tell a closed database from refused credentials; the kinds list each reason. Durable state and freshness gauges shipped by the operator itself are tracked in issue #226.

Plan retention

Plans are owned by the schema, and a plan’s chunks by the plan. Old plan objects may remain useful as audit evidence until Kubernetes garbage collection removes the owning schema; only the exact UID and fingerprint in status.plan are current. Completed Jobs receive a cleanup TTL before the controller clears the active operation: five minutes, or 24 hours for an Apply that failed or whose outcome is unknown, when a person approved it or its migration run stopped the resource. Failure to schedule that TTL leaves the operation retryable. A guard that refuses the 24 hours, as the guard of the previous release does for a moment during an upgrade, gets the five minutes instead, with a FailedApplyLogShortened Warning Event, and the resource says which the Job holds. The exact current Apply may receive its TTL while still running when the controller must persist an uncertain outcome; the countdown starts only after the Job finishes. During schema deletion, a terminal Job that cannot satisfy the exact cleanup admission contract is left to owner garbage collection instead of blocking the finalizer on a weaker write.

An executable plan is limited to 8 MiB, including its trailing newline. The runner rejects a larger native plan before publication. Accepted bytes are stored in immutable 512 KiB PtahSchemaPlanChunk objects; that chunk size leaves headroom below the Kubernetes object-size limit after API JSON base64 encoding. A plan that reaches an Apply is copied into immutable ConfigMaps of the same size and names, owned by the plan, which is what the Apply Pod mounts.

The chunks are storage, not a reading interface. kubectl ptah plan <schema> reads a stored plan back the way the operator does – every chunk against its digest and size, joined in index order, the whole document against its content digest – and --applied reads the one the last confirmed apply ran. See Read a plan, which carries the install.

Pruning stored plans

A plan is never overwritten. A new fingerprint publishes a new PtahSchemaPlan or PtahMigrationPlan and its chunks, and the old ones stay until their owner is deleted. Storage therefore grows with every distinct plan: at the 8 MiB ceiling, a hundred retained plans hold 800 MiB of SQL before metadata. That is arithmetic about the ceiling, not a measurement of any installation.

The operator prunes nothing on its own. How much history to keep is a decision about your audit requirements and your etcd budget, and neither is something this project can choose for you. What it can say exactly is which plans are not safe to delete.

Which plans are pinned

A plan is pinned while any of these names it. Each is a field on a live object, so the set is checkable with kubectl rather than inferred:

Field	Why the plan must stay
ptahschema.status.plan	published and awaiting approval, or being applied
ptahschema.status.pendingObservation.plan	the post-Apply verification owed for it has not finished
ptahschema.status.applied.planRef	the evidence of what the last confirmed apply ran
ptahschema.status.pendingBindingRetirement.plan	an execution-binding rotation retired it, and the approvals that name it are still being marked stale
ptahschemaapproval.spec.planRef	a person authorized these exact bytes
ptahmigrationapproval.spec.planRef	the same, for a migration sequence
ptahmigration.status.plan	published and awaiting approval, or being applied
ptahmigration.status.activeOperation.planRef	an Apply is in flight against it
ptahmigration.status.unresolvedRun.planRef	a run nobody accounted for may have executed it
ptahmigration.status.lastRun.planRef	the evidence of what the last run carried out

A list of pins from before 0.2.10 leaves status.lastRun.planRef out, so a prune that used one can delete the plan that field names. The operator goes on working, and the field names bytes nothing holds: the check below reports it as a pinned plan that was deleted, and the retention audit in hack/capacity refuses it as a missing plan.

The unresolvedRun row is the one that matters most and is easiest to miss. An unresolved run is a migration that may already have changed the database, and its plan is what a person reads to find out what it would have done. Deleting it destroys the only record of the work in question.

Prune plans that nothing pins

Before you start

You need read access to both families and their approvals across the namespaces you are pruning, and delete access to plan objects in them. Nothing cluster-scoped is touched.

Read Which plans are pinned first. Every pin is a field on a live object, so the set is checkable rather than inferred, and the row easiest to miss is the one that matters most: a plan named by ptahmigration.status.unresolvedRun.planRef is the only record of work nobody has accounted for.

Do this while no operation is in flight for the resources involved. A plan that is not pinned when you list it can become pinned a moment later, which is the same race any external pruner has and the reason the window matters.

Export anything you intend to keep as evidence before deleting it. Nothing reconstructs those bytes once the chunks are gone, and Export a plan before deleting it carries the commands and the digest check that says the export is the plan.

Run it

Collect the pinned names first, then delete plans outside that set:

kubectl get ptahschemas,ptahmigrations -A -o json |
  jq -r '.items[] | .status as $s |
    ($s.plan.name // empty),
    ($s.pendingObservation.plan.name // empty),
    ($s.applied.planRef.name // empty),
    ($s.pendingBindingRetirement.plan.name // empty),
    ($s.activeOperation.planRef.name // empty),
    ($s.unresolvedRun.planRef.name // empty),
    ($s.lastRun.planRef.name // empty)' |
  sort -u > pinned.txt
kubectl get ptahschemaapprovals,ptahmigrationapprovals -A \
  -o jsonpath='{range .items[*]}{.spec.planRef.name}{"\n"}{end}' |
  sort -u >> pinned.txt

Delete a plan only if its name is absent from pinned.txt. Its chunks, and any ConfigMaps an Apply projected it into, are owned by the plan, so garbage collection removes them after it. Delete one plan at a time with --cascade=foreground --wait=true, and wait for its payloads to disappear before deleting the next. This bounds the garbage-collection work competing with live API reads during maintenance. Each payload carries the label operator.ptah.run/plan=<plan-name>, which is how to confirm they went.

What proves it worked

The plans you chose are gone with their chunks, and the collection above, re-run, still resolves every name it returns. A name that no longer resolves is the finding: a pinned plan was deleted, and the resource that pinned it now names bytes nothing holds.

Where to stop

Absence from pinned.txt is the only permission to delete a plan. Stop if the plan is named by status.unresolvedRun.planRef, whatever else is true of it: that is the record of a run that may already have changed a database, and deleting it destroys the only description of the work in question. Stop if an export’s digest disagrees with what the plan records – a chunk is already missing or was tampered with, which is a finding rather than a pruning problem.

If it fails

The collection reads and nothing else, so a run that fails partway is repeated rather than repaired. A deletion is the other way around: nothing restores a plan or its chunks, and an export taken beforehand is the whole of the recovery path. Take one for anything you are not certain about, and check its digest before the delete rather than after.

Export a plan before deleting it

kubectl ptah plan reads the current plan and --applied the last confirmed one. Neither is the plan you are about to prune – those two are pinned – so a historical plan is exported from its own objects.

What makes an exported plan interpretable later travels with it. The plan document carries every binding it was computed under: the fingerprint an approval names, the artifact and target identity digests, the policy, the execution binding, and the manager, runner and executor identities that would have run it. The chunks carry the SQL. The approval, where one exists, carries who decided and when.

NS=<namespace>
PLAN=<plan-name>
kubectl -n "$NS" get ptahschemaplan "$PLAN" -o json > "$PLAN.plan.json"
jq -r '.spec.chunks | sort_by(.index)[] | .name' "$PLAN.plan.json" |
  while read -r chunk; do
    kubectl -n "$NS" get ptahschemaplanchunk "$chunk" -o json |
      jq -r '.spec.data' | base64 -d
  done > "$PLAN.sql"
kubectl -n "$NS" get ptahschemaapprovals -o json |
  jq --arg plan "$PLAN" '[.items[] | select(.spec.planRef.name == $plan)]' \
  > "$PLAN.approvals.json"

Check what you exported against what the plan says it is. The chunks are ordered and digested individually, and the plan records the digest of the bytes they reconstruct:

printf 'exported sha256:%s\n' "$(shasum -a 256 "$PLAN.sql" | cut -d' ' -f1)"
jq -r '"recorded  " + .spec.contentDigest' "$PLAN.plan.json"

The two must match. A mismatch means a chunk is already missing or was tampered with, and the export is not the plan – find out why before deleting anything.

A PtahMigrationPlan needs no chunk step: it stores no SQL, only the ordered versions and their checksums, and the statements stay in the artifact its spec.artifactDigest pins. Export the plan document and keep the artifact.

Kubernetes versions

The supported minor window and update procedure are defined in Kubernetes support. A support-window change adds the new minor and removes the oldest minor atomically, after the entire real-cluster matrix succeeds.