PtahSuspend and resume
One field stops reconciliation without deleting anything, and resuming picks up where it stopped.
What it shows. How to take a database out of the operator's hands during an incident, and give it back.
Where it starts. Where Apply a schema ends: storefront converged on schemas/v1.sql, published as an artifact, under apply Always.
A replay of a verified run, not a live cluster. 12 checks held while it was recorded. Play types it out in the frame below; the transcript is the same session, already there.
# Converged, and reconciliation is active.$ kubectl ptah schema storefront -n "$NAMESPACE"Schema: demo/storefrontPhase: InSyncReady: True, InSync: Schema is in syncPolicy: apply Always, allowDestructive false, driftSeverity allArtifact: oci://e2e-registry.demo.svc.cluster.local:5000/schemas/demo@sha256:8763787589af7fc2262f3f6b520ef5cae57ba91fb7aaf19e5ead8c5ae75e3d6fVerified: trueObserved: <time>Drift: noneApplied: ptah-plan-2d431045c915ef2fc7d3c841 at <time> No plan is published. Approvals: none Conditions: TYPE STATUS REASON SINCE MESSAGE EngineSupported True SupportedEngine <time> Database engine "PostgreSQL" is supported Ready True InSync <time> Schema is in sync ReconciliationFailed False Succeeded <time> The latest operation completed ArtifactResolved True DigestPinned <time> Desired OCI reference resolved to immutable content ArtifactVerified True PolicySatisfied <time> Artifact type and verification policy were satisfied DatabaseReachable True Observed <time> Database schema was observed DriftDetected False ScopedConverged <time> The authoritative managed scope has no changes InSync True ScopedConverged <time> A stable scoped plan proves convergence ApprovalRequired False Satisfied <time> Current plan approval requirements are satisfied PlanReady False NoChanges <time> No executable plan is required Applying False JobCompleted <time> Apply Job completed; convergence observation is pending Suspended False Active <time> Reconciliation is active kubectl events -n demo --for ptahschema/storefront prints its Events. # Suspend it. One field, and the resource stays exactly where it is.$ kubectl ptah suspend schema/storefront -n "$NAMESPACE" --waitSuspend requested for ptahschema/storefront in namespace demo. Requested: spec.suspend true, generation 2 Observed: generation 1, phase InSyncThe operator starts no new Job once it observes generation 2. Suspension cancels no SQL.ptahschema/storefront in namespace demo reached suspended. Phase: Suspended Generation: 2, observed by the operator Database: last read at <time>kubectl ptah schema storefront -n demo shows where it stands. # Now publish a change it would otherwise pick up, and point the resource at it.$ REF=$(ptah schema push "oci://$PTAH_OCI_REGISTRY/schemas/demo:v2-$$" \ --schema-file demo/schemas/v2.sql --dialect postgres --plain-http --format reference)kubectl ptah set-source schema/storefront -n "$NAMESPACE" \ --oci "oci://$REGISTRY_IN_CLUSTER/${REF#"oci://$PTAH_OCI_REGISTRY/"}"Source set for ptahschema/storefront in namespace demo. Requested: spec.desired.ociRef oci://e2e-registry.demo.svc.cluster.local:5000/schemas/demo@sha256:a87c0faa75836d3a793d68b69d386fe2bf9c7af21c79b289f28630d826c66b5c, generation 3 Replaced: oci://e2e-registry.demo.svc.cluster.local:5000/schemas/demo@sha256:8763787589af7fc2262f3f6b520ef5cae57ba91fb7aaf19e5ead8c5ae75e3d6f Kept: verificationPolicyFrom demo-verification-policy key policy.yaml, registryAuthFrom demo-registry, transport.plainHTTP true Observed: generation 2, phase Suspended Resolved: oci://e2e-registry.demo.svc.cluster.local:5000/schemas/demo@sha256:8763787589af7fc2262f3f6b520ef5cae57ba91fb7aaf19e5ead8c5ae75e3d6f, verified Apply: spec.policy.apply AlwaysThe resource is suspended: the operator starts no Job until it is resumed, so nothing resolves the new reference before then, and a wait for its plan lasts until then too. After kubectl ptah resume schema/storefront -n demo the operator resolves the new reference, checks it against the verification policy, reads the database and plans from what it finds. A plan or an approval from the previous artifact does not carry over.Under spec.policy.apply Always the operator applies that plan without a person, unless it is destructive or changes privileges: those wait for an approval.set-source --wait waits in the same command until a plan from the new reference waits for an approval or the resource rests in sync from it; kubectl ptah wait, run later, counts only a reading that begins after it does. # Nothing happens, which is the point. A wait gives up and names what holds it.$ kubectl ptah wait schema/storefront -n "$NAMESPACE" --for in-sync --timeout 30sWaiting up to 30s for ptahschema/storefront in namespace demo to reach in-sync.kubectl-ptah: ptahschema/storefront in namespace demo did not reach in-sync within 30s. Phase: Suspended Held by: Suspended=True, Requested: New database operations are suspended Waiting for: a read of the database that began after the wait did (the last one is from <time>)kubectl ptah schema storefront -n demo shows every condition with its message. # The database is untouched.$ kubectl -n "$NAMESPACE" exec deploy/demo-psql -- psql -c "\dt" List of relations Schema | Name | Type | Owner--------+-----------+-------+--------------- public | customers | table | ptah_external(1 row) # Resume, and it starts from what is actually there rather than from what it remembered.$ kubectl ptah resume schema/storefront -n "$NAMESPACE" --waitResume requested for ptahschema/storefront in namespace demo. Requested: spec.suspend false, generation 4 Observed: generation 3, phase SuspendedOnce the operator observes generation 4 it reads the database again and decides from what it finds.ptahschema/storefront in namespace demo reached in-sync. Phase: InSync Generation: 4, observed by the operator Database: last read at <time>kubectl ptah schema storefront -n demo shows where it stands. # It planned against the live database and converged on the revision it was given.$ kubectl -n "$NAMESPACE" exec deploy/demo-psql -- psql -c "\dt" List of relations Schema | Name | Type | Owner--------+-----------+-------+--------------- public | customers | table | ptah_external public | orders | table | ptah_external(2 rows)
What was checked
Every step states what has to hold before its output may be published. A condition is read as its type, status and reason on the live object, never as a phrase in a message.
| Step | Claim | Held |
|---|---|---|
| 1 | exits 0, prints "InSync", prints "Suspended", prints "Active" | yes |
| 1 | ptahschema/storefront: Suspended=False (Active) | yes |
| 2 | exits 0, prints "Suspend requested", prints "spec.suspend true", prints "reached suspended" | yes |
| 2 | ptahschema/storefront: Suspended=True (Requested), status.source.digest is set, kept as resolved, status.target.lastObservedAt is set, kept as observed, status.applied.planFingerprint is set, kept as applied | yes |
| 3 | exits 0, prints "Source set for", prints "The resource is suspended: the operator starts no Job until it is resumed", prints "kubectl ptah resume schema/storefront", never prints "Once the operator observes generation" | yes |
| 3 | ptahschema/storefront: metadata.generation is set, kept as pointed | yes |
| 4 | exits 7, reports "did not reach in-sync", reports "Phase: Suspended", reports "Held by: Suspended=True, Requested", never reports "Last read", never reports "deadline exceeded" | yes |
| 4 | ptahschema/storefront: Suspended=True (Requested), status.observedGeneration is the value kept as pointed, status.source.digest is the value kept as resolved, status.target.lastObservedAt is the value kept as observed, status.applied.planFingerprint is the value kept as applied, status.activeOperation is unset | yes |
| 5 | exits 0, prints "customers", never prints "orders" | yes |
| 6 | exits 0, prints "Resume requested", prints "reached in-sync" | yes |
| 6 | ptahschema/storefront: InSync=True (ScopedConverged) | yes |
| 7 | exits 0, prints "customers", prints "orders" | yes |
What it ran against
- Kubernetes1.37.0
- Operator42522be3fd5a
- Ptahv0.17.0
- Executorsha256:181e27972091
- Recorded2026-11-02
- Scenarios at42522be3fd5a
What the commands read
A session is repeated, not admired, so every name in it is one your own environment can carry. There is nothing else: no helper of ours stands between a command and the cluster, and the recorder refuses a step that reads a name this list does not hold. The commands arekubectl, ptah and kubectl ptah, each of which you install once; the last one is the operator’s own client. They run from the root of a clone of this repository, which holds the files they name: the SQL under demo/schemas, demo/migrations,demo/migrations-checkpoint and demo/mysql, the reference data underdemo/reference, and the manifests under demo/manifests.
- KUBECONFIGthe cluster the commands run against
- NAMESPACEthe namespace the resources and their database credentials live in
- OPERATOR_NAMESPACEwhere the chart installed the operator
- CONTROLLERthe controller Deployment’s name, from that release
- EXECUTOR_IMAGEthe executor image the chart runs, its
executorImagevalue - REGISTRY_IN_CLUSTERthe registry address a Pod in the cluster resolves
- PTAH_OCI_REGISTRYthe registry address the push goes to
- PTAH_OCI_USERNAMEthe account that push uses, read by
ptahitself - PTAH_OCI_PASSWORDits password, which no command prints
The last two are how a CI job hands ptah a registry credential. On a workstation,ptah oci login stores it once instead, reading the password from a prompt or from standard input rather than from the command line.
The registry has two addresses, the way a registry in an ordinary deployment does: a push goes to PTAH_OCI_REGISTRY, and the resource names REGISTRY_IN_CLUSTER with the same repository and digest. The registry Secret a resource names says which address its credential is for, and the operator sends it to no other.
The lab itself appears in one command, demo/bin/lab versions in Stand up the lab, which says what the lab is made of. What a run starts from is its preparation, which is not recorded; Where it starts above says what that is and which run ends there. Rundemo/acceptance/reproduce.sh to watch Approve a plan repeated from a directory where the lab’s scripts do not exist, by an account that may not create a Job, with the plan approved by a second account that may create neither a schema nor a Job, and then the same schema published with ptah schema push and moved to its next revision withkubectl ptah set-source.
The same changes, as edits
Each kubectl ptah command that changes a resource changes one field of it, so the same change is an ordinary edit, with kubectl edit ptahschema storefront -n "$NAMESPACE"or in a manifest you apply; the runs that keep their resources as files underdemo/manifests make their changes that way. What the commands add is checking: set-source takes only a reference pinned by digest, changes nothing while an Apply may be running, and changes nothing another writer changed since it read the resource.
| Command | The line it changes |
|---|---|
kubectl ptah suspend | spec.suspend: true |
kubectl ptah resume | spec.suspend: false |
kubectl ptah set-source --oci | spec.desired.ociRef of a schema, spec.artifact.ociRef of a migration |
kubectl ptah refresh | the annotation operator.ptah.run/read-requested-at, set to the time of the request |
An approval is not one of them. It is an object of its own, created once by the person who decided: kubectl ptah approve --dry-run client prints it as YAML for a repository, to be created with kubectl create, and never edited afterwards.
The scenario is demo/scenarios/suspend-resume.yaml, and make demo replays the whole set against a lab of your own. All recorded runs.