PtahFail, then recover
Point it at a database that is not there. It says so, keeps trying, and converges when it is.
What it shows. What a failure looks like in the status, and that recovery needs no intervention beyond the fix.
Where it starts. An empty database, and a Secret named demo-database-broken that the lab planted as the fault: a URL, with a user and password, for a host that does not exist. The schema is read from a ConfigMap, the delivery a team uses when its manifests already carry the files.
A replay of a verified run, not a live cluster. 10 checks held while it was recorded. Play types it out in the frame below; the transcript is the same session, already there.
# The schema as a ConfigMap, and a resource pointed at a Secret whose database does not answer.$ kubectl -n "$NAMESPACE" create configmap storefront-v1 --from-file=schema.sql=demo/schemas/v1.sqlkubectl -n "$NAMESPACE" apply -f demo/manifests/failure-recovery/storefront-unreachable.yamlconfigmap/storefront-v1 createdptahschema.operator.ptah.run/storefront created # Wait for the operator to give its verdict.$ kubectl -n "$NAMESPACE" wait ptahschema/storefront --for=condition=ReconciliationFailed --timeout=6mptahschema.operator.ptah.run/storefront condition met # It failed, with a reason a machine can read. The message is only the exit code of ptah.$ kubectl ptah schema storefront -n "$NAMESPACE"Schema: demo/storefrontPhase: FailedReady: False, OperationFailed: Reconciliation did not completeReconciliationFailed: True, OperationFailed: child_exit: ptah exited with code 2Policy: apply Always, allowDestructive false, driftSeverity allSource: ConfigMap storefront-v1Snapshot: ptah-src-f5ad43883ac4c9a2dee28890Content: sha256:5a00a478af9f3c4a52e37aff33e574f580bcb6b94eb68860feb483b509b09667Observed: not yet No plan is published. Approvals: none Conditions: TYPE STATUS REASON SINCE MESSAGE EngineSupported True SupportedEngine <time> Database engine "PostgreSQL" is supported Suspended False Active <time> Reconciliation is active ReconciliationFailed True OperationFailed <time> child_exit: ptah exited with code 2 ArtifactResolved True SnapshotTaken <time> ConfigMap storefront-v1 was copied into snapshot ptah-src-f5ad43883ac4c9a2dee28890 Ready False OperationFailed <time> Reconciliation did not complete kubectl events -n demo --for ptahschema/storefront prints its Events. # A failure is an event too, so it is in the record rather than only in a log.$ kubectl events -n "$NAMESPACE" --for ptahschema/storefrontLAST SEEN TYPE REASON OBJECT MESSAGE11s Normal OperationStarted PtahSchema/storefront Observe Job ptah-observe-storefront-d4a394b4e6ca5d45 started1s Warning OperationFailed PtahSchema/storefront child_exit: ptah exited with code 2 # Verification: no credential in the status, and the check reads all of it.$ status=$(kubectl -n "$NAMESPACE" get ptahschema storefront -o jsonpath='{.status}') || status=case $status in '{'*'}') ;; *) printf 'could not read the status, so nothing was checked\n' >&2 exit 1 ;;esacfor planted in 'nobody:nobody' 'no-such-database.demo.svc.cluster.local'; do if printf '%s' "$status" | grep -qF "$planted"; then printf 'the status carries the credential this scenario planted\n' >&2 exit 1 fidoneprintf 'checked the whole status, %s bytes, no credential in it\n' \ "$(printf '%s' "$status" | wc -c | tr -d ' ')"checked the whole status, 2864 bytes, no credential in it # The fix is one line of the manifest. Apply it.$ diff -u demo/manifests/failure-recovery/storefront-unreachable.yaml demo/manifests/failure-recovery/storefront.yamlkubectl -n "$NAMESPACE" apply -f demo/manifests/failure-recovery/storefront.yaml--- demo/manifests/failure-recovery/storefront-unreachable.yaml 2026-11-02 14:04:31.814213204 +0000+++ demo/manifests/failure-recovery/storefront.yaml 2026-11-02 14:04:31.814354479 +0000@@ -10,7 +10,7 @@ engine: PostgreSQL coordinationKey: demo/storefront/primary urlFrom:- name: demo-database-broken+ name: demo-database key: url desired: configMap:ptahschema.operator.ptah.run/storefront configured # It retries on its own interval and converges. Nothing was restarted by hand.$ kubectl ptah wait schema/storefront -n "$NAMESPACE" --for in-syncptahschema/storefront in namespace demo reached in-sync. Phase: InSync Generation: 2, observed by the operator Database: last read at <time>kubectl ptah schema storefront -n demo shows where it stands. # And the database has the table.$ kubectl -n "$NAMESPACE" exec deploy/demo-psql -- psql -c "\dt" List of relations Schema | Name | Type | Owner--------+-----------+-------+--------------- public | customers | table | ptah_external(1 row)
What was checked
Every step states what has to hold before its output may be published. A condition is read as its type, status and reason on the live object, never as a phrase in a message.
| Step | Claim | Held |
|---|---|---|
| 1 | exits 0, prints "configmap/storefront-v1 created", prints "storefront created" | yes |
| 2 | exits 0, prints "condition met" | yes |
| 2 | ptahschema/storefront: ReconciliationFailed=True (OperationFailed) | yes |
| 3 | exits 0, prints "Failed", prints "ReconciliationFailed", prints "OperationFailed", prints "ptah exited with code", never prints "nobody:nobody", never prints "no-such-database" | yes |
| 4 | exits 0, prints "storefront", prints "OperationFailed", never prints "nobody:nobody", never prints "no-such-database" | yes |
| 5 | exits 0, prints "checked the whole status", prints "no credential in it", never prints "nobody:nobody" | yes |
| 6 | exits 0, prints "- name: demo-database-broken", prints "+ name: demo-database", prints "storefront configured" | yes |
| 7 | exits 0, prints "reached in-sync" | yes |
| 7 | ptahschema/storefront: ReconciliationFailed=False (Succeeded) | yes |
| 8 | exits 0, prints "customers" | yes |
What it ran against
- Kubernetes1.37.0
- Operator42522be3fd5a
- Ptahv0.17.0
- Executorsha256:181e27972091
- Recorded2026-11-02
- Scenarios at42522be3fd5a
What the commands read
A session is repeated, not admired, so every name in it is one your own environment can carry. There is nothing else: no helper of ours stands between a command and the cluster, and the recorder refuses a step that reads a name this list does not hold. The commands arekubectl, ptah and kubectl ptah, each of which you install once; the last one is the operator’s own client. They run from the root of a clone of this repository, which holds the files they name: the SQL under demo/schemas, demo/migrations,demo/migrations-checkpoint and demo/mysql, the reference data underdemo/reference, and the manifests under demo/manifests.
- KUBECONFIGthe cluster the commands run against
- NAMESPACEthe namespace the resources and their database credentials live in
- OPERATOR_NAMESPACEwhere the chart installed the operator
- CONTROLLERthe controller Deployment’s name, from that release
- EXECUTOR_IMAGEthe executor image the chart runs, its
executorImagevalue - REGISTRY_IN_CLUSTERthe registry address a Pod in the cluster resolves
- PTAH_OCI_REGISTRYthe registry address the push goes to
- PTAH_OCI_USERNAMEthe account that push uses, read by
ptahitself - PTAH_OCI_PASSWORDits password, which no command prints
The last two are how a CI job hands ptah a registry credential. On a workstation,ptah oci login stores it once instead, reading the password from a prompt or from standard input rather than from the command line.
The registry has two addresses, the way a registry in an ordinary deployment does: a push goes to PTAH_OCI_REGISTRY, and the resource names REGISTRY_IN_CLUSTER with the same repository and digest. The registry Secret a resource names says which address its credential is for, and the operator sends it to no other.
The lab itself appears in one command, demo/bin/lab versions in Stand up the lab, which says what the lab is made of. What a run starts from is its preparation, which is not recorded; Where it starts above says what that is and which run ends there. Rundemo/acceptance/reproduce.sh to watch Approve a plan repeated from a directory where the lab’s scripts do not exist, by an account that may not create a Job, with the plan approved by a second account that may create neither a schema nor a Job, and then the same schema published with ptah schema push and moved to its next revision withkubectl ptah set-source.
The same changes, as edits
Each kubectl ptah command that changes a resource changes one field of it, so the same change is an ordinary edit, with kubectl edit ptahschema storefront -n "$NAMESPACE"or in a manifest you apply; the runs that keep their resources as files underdemo/manifests make their changes that way. What the commands add is checking: set-source takes only a reference pinned by digest, changes nothing while an Apply may be running, and changes nothing another writer changed since it read the resource.
| Command | The line it changes |
|---|---|
kubectl ptah suspend | spec.suspend: true |
kubectl ptah resume | spec.suspend: false |
kubectl ptah set-source --oci | spec.desired.ociRef of a schema, spec.artifact.ociRef of a migration |
kubectl ptah refresh | the annotation operator.ptah.run/read-requested-at, set to the time of the request |
An approval is not one of them. It is an object of its own, created once by the person who decided: kubectl ptah approve --dry-run client prints it as YAML for a repository, to be created with kubectl create, and never edited afterwards.
The scenario is demo/scenarios/failure-recovery.yaml, and make demo replays the whole set against a lab of your own. All recorded runs.