PtahFail, then recover
Point it at a database that is not there. It says so, keeps trying, and converges when it is.
What it shows. What a failure looks like in the status, and that recovery needs no intervention beyond the fix.
A replay of a verified run, not a live cluster. 9 checks held while it was recorded. Play types it out in the frame below; the transcript is the same session, already there.
# The same resource, pointed at a database that does not answer.$ APPLY=Always demo/bin/lab manifest storefront "$(cat demo/.lab/digest)" \ | sed 's/name: demo-database/name: demo-database-broken/' \ | kubectl apply -f -ptahschema.operator.ptah.run/storefront created # It reports the database as unreachable, with a reason a machine can read.$ kubectl -n "$NAMESPACE" get ptahschema storefront -o json \ | jq -r '.status.phase, (.status.conditions[] | select(.type == "ReconciliationFailed") | .message)'Failedchild_exit: ptah exited with code 2 # A failure is an event too, so it is in the record rather than only in a log.$ kubectl -n "$NAMESPACE" get events --field-selector involvedObject.name=storefront --sort-by=.lastTimestamp | tail -510s Normal OperationCompleted ptahschema/storefront resolve operation completed10s Normal OperationStarted ptahschema/storefront Verify Job ptah-verify-storefront-6ac98bb7e80db721 started7s Normal OperationCompleted ptahschema/storefront verify operation completed7s Normal OperationStarted ptahschema/storefront Observe Job ptah-observe-storefront-fbbae8adf9a75b68 started2s Warning OperationFailed ptahschema/storefront child_exit: ptah exited with code 2 # No credential in any of it, and the check reads all of it.$ status=$(kubectl -n "$NAMESPACE" get ptahschema storefront -o json | jq -ce '.status | objects') || { printf 'could not read the status, so nothing was checked\n' >&2 exit 1}for planted in 'nobody:nobody' 'no-such-database.demo.svc.cluster.local'; do if printf '%s' "$status" | grep -qF "$planted"; then printf 'the status carries the credential this scenario planted\n' >&2 exit 1 fidoneprintf 'checked the whole status, %s bytes, no credential in it\n' \ "$(printf '%s' "$status" | wc -c | tr -d ' ')"printf '%s' "$status" | jq -r ' "phase " + .phase, "reason " + (.conditions[] | select(.type == "ReconciliationFailed") | .reason)'checked the whole status, 3444 bytes, no credential in itphase Failedreason OperationFailed # Now fix the one thing that was wrong.$ kubectl -n "$NAMESPACE" patch ptahschema storefront --type=merge \ -p '{"spec":{"target":{"urlFrom":{"name":"demo-database","key":"url"}}}}'ptahschema.operator.ptah.run/storefront patched # It retries on its own interval and converges. Nothing was restarted by hand.$ kubectl -n "$NAMESPACE" exec deploy/demo-psql -- psql -c "\dt" List of relations Schema | Name | Type | Owner--------+-----------+-------+--------------- public | customers | table | ptah_external(1 row)
What was checked
Every step states what has to hold before its output may be published. A condition is read as its type, status and reason on the live object, never as a phrase in a message.
| Step | Claim | Held |
|---|---|---|
| 1 | exits 0, prints "created" | yes |
| 2 | ptahschema/storefront: ReconciliationFailed=True (OperationFailed) | yes |
| 2 | exits 0, prints "Failed" | yes |
| 3 | exits 0, prints "storefront" | yes |
| 4 | exits 0, prints "checked the whole status", prints "no credential in it", prints "OperationFailed", never prints "nobody:nobody" | yes |
| 5 | exits 0, prints "patched" | yes |
| 6 | ptahschema/storefront: InSync=True (ScopedConverged) | yes |
| 6 | exits 0, prints "customers" | yes |
| 6 | ptahschema/storefront: ReconciliationFailed=False (Succeeded) | yes |
What it ran against
- Kubernetes1.37.0
- Operatorc08fc73ebbae
- Ptahf6e562c
- Executorsha256:137662e6db45
- Recorded2026-09-28
- Scenarios atc08fc73ebbae
What the commands read
A session is repeated, not admired, so every name in it is one your own environment can carry. There is nothing else: no helper of ours stands between a command and the cluster, and the recorder refuses a step that reads a name this list does not hold. The commands arekubectl, ptah and kubectl ptah, each of which you install once; the last one is the operator’s own read-only client.
- KUBECONFIGthe cluster the commands run against
- NAMESPACEthe namespace the schema and its database credentials live in
- OPERATOR_NAMESPACEwhere the chart installed the operator
- CONTROLLERthe controller Deployment’s name, from that release
- REGISTRY_IN_CLUSTERthe registry address a Pod in the cluster resolves
- PTAH_OCI_REGISTRYthe registry address the push goes to
- PTAH_OCI_USERNAMEthe account that push uses
- PTAH_OCI_PASSWORDits password, which no command prints
A session also uses demo/bin/lab in two places, and both are the lab handing over what it generated rather than doing the demonstration’s work:lab credentials prints the three registry values above, and lab manifestfills the published template with values the step states. Rundemo/acceptance/reproduce.sh to watch the same scenario repeated from a directory where neither exists, by an account that may not create a Job, with the plan approved by a second account that may create neither a schema nor a Job.
The scenario is demo/scenarios/failure-recovery.yaml, and make demo replays the whole set against a lab of your own. All recorded runs.