PtahFail, then recover
Point it at a database that is not there. It says so, keeps trying, and converges when it is.
What it shows. What a failure looks like in the status, and that recovery needs no intervention beyond the fix.
A replay of a verified run, not a live cluster. 9 checks held while it was recorded. Play types it out in the frame below; the transcript is the same session, already there.
# The same resource, pointed at a database that does not answer.$ APPLY=Always demo/bin/lab manifest storefront "$(cat demo/.lab/digest)" \ | sed 's/name: demo-database/name: demo-database-broken/' \ | kubectl apply -f -ptahschema.operator.ptah.run/storefront created # It reports the database as unreachable, with a reason a machine can read.$ kubectl -n "$NAMESPACE" get ptahschema storefront -o json \ | jq -r '.status.phase, (.status.conditions[] | select(.type == "ReconciliationFailed") | .message)'Failedchild_exit: ptah exited with code 2 # A failure is an event too, so it is in the record rather than only in a log.$ kubectl -n "$NAMESPACE" get events --field-selector involvedObject.name=storefront --sort-by=.lastTimestamp | tail -512s Normal OperationCompleted ptahschema/storefront resolve operation completed11s Normal OperationStarted ptahschema/storefront Verify Job ptah-verify-storefront-71fd1b934301ac75 started7s Normal OperationCompleted ptahschema/storefront verify operation completed6s Normal OperationStarted ptahschema/storefront Observe Job ptah-observe-storefront-e4d00056326601d7 started1s Warning OperationFailed ptahschema/storefront child_exit: ptah exited with code 2 # No credential in any of it. This is the whole status the operator wrote.$ kubectl -n "$NAMESPACE" get ptahschema storefront -o json | jq -r '.status | tostring' | head -c 400{"activeOperation":{"attempt":2,"coordinationDigest":"sha256:1f99f1e0ceb341a5b6ca5da75161513672b13c811fc50e1725fc7dec0f8aa99f","executionBindingID":"v1-b50602bd8c7b04187031ee43aee0f6eb","id":"sha256:81dabcf8d2a725da080ba4e18858d7cdcfd5559b6c869f5a045a6a0e37461234","inputFingerprint":"sha256:4aac580d2872b7e2b6763643e8a9bff7b55ae361030a2c5e7d6776c6ef6ca587","jobName":"ptah-observe-storefront-ea3c44d # Now fix the one thing that was wrong.$ kubectl -n "$NAMESPACE" patch ptahschema storefront --type=merge \ -p '{"spec":{"target":{"urlFrom":{"name":"demo-database","key":"url"}}}}'ptahschema.operator.ptah.run/storefront patched # It retries on its own interval and converges. Nothing was restarted by hand.$ kubectl -n "$NAMESPACE" exec deploy/demo-psql -- psql -c "\dt" List of relations Schema | Name | Type | Owner--------+-----------+-------+--------------- public | customers | table | ptah_external(1 row)
What was checked
Every step states what has to hold before its output may be published. A condition is read as its type, status and reason on the live object, never as a phrase in a message.
| Step | Claim | Held |
|---|---|---|
| 1 | exits 0, prints "created" | yes |
| 2 | ptahschema/storefront: ReconciliationFailed=True (OperationFailed) | yes |
| 2 | exits 0, prints "Failed" | yes |
| 3 | exits 0, prints "storefront" | yes |
| 4 | exits 0, never prints "nobody:nobody" | yes |
| 5 | exits 0, prints "patched" | yes |
| 6 | ptahschema/storefront: InSync=True (ScopedConverged) | yes |
| 6 | exits 0, prints "customers" | yes |
| 6 | ptahschema/storefront: ReconciliationFailed=False (Succeeded) | yes |
What it ran against
- Kubernetes1.37.0
- Operator8c7fe187707f
- Ptahv0.6.0
- Executorsha256:91ec0a6fa27f
- Recorded2026-09-17
- Scenarios at8c7fe187707f
What the commands read
A session is repeated, not admired, so every name in it is one your own environment can carry. There is nothing else: no helper of ours stands between a command and the cluster, and the recorder refuses a step that reads a name this list does not hold. The commands arekubectl, ptah and kubectl ptah, each of which you install once; the last one is the operator’s own read-only client.
- KUBECONFIGthe cluster the commands run against
- NAMESPACEthe namespace the schema and its database credentials live in
- OPERATOR_NAMESPACEwhere the chart installed the operator
- CONTROLLERthe controller Deployment’s name, from that release
- REGISTRY_IN_CLUSTERthe registry address a Pod in the cluster resolves
- PTAH_OCI_REGISTRYthe registry address the push goes to
- PTAH_OCI_USERNAMEthe account that push uses
- PTAH_OCI_PASSWORDits password, which no command prints
A session also uses demo/bin/lab in two places, and both are the lab handing over what it generated rather than doing the demonstration’s work:lab credentials prints the three registry values above, and lab manifestfills the published template with values the step states. Rundemo/acceptance/reproduce.sh to watch the same scenario repeated from a directory where neither exists, by an account that may not create a Job.
The scenario is demo/scenarios/failure-recovery.yaml, and make demo replays the whole set against a lab of your own. All recorded runs.