PtahRecover a MySQL migration that stopped halfway
DDL that committed, a run nobody can retry, and a repair decided by what the database holds.
What it shows. How to find out what a partial MySQL migration left behind before choosing how to repair it.
A replay of a verified run, not a live cluster. 21 checks held while it was recorded. Play types it out in the frame below; the transcript is the same session, already there.
# Release 1 creates the table on the lab's MySQL, and this resource applies it on its own.$ APPLY=Always demo/bin/lab manifest deliveries "$(cat demo/.lab/deliveries-1-digest)" | kubectl apply -f -ptahmigration.operator.ptah.run/deliveries created # Three deliveries arrive. The next release has to keep them.$ kubectl -n "$NAMESPACE" exec deploy/demo-mysql-client -- mysql demo -e \ "INSERT INTO deliveries VALUES (1, 'D-100'), (2, 'D-200'), (3, 'D-300'); SELECT * FROM deliveries"id reference1 D-1002 D-2003 D-300 # Release 2 adds a status, fills it, then constrains it, naming a column that is not there.$ cat demo/mysql/release-2/0000000002_add_status.up.sql-- Adds a status, fills it, and then constrains it. The constraint names a-- column that does not exist, so the third statement fails after the first two-- have committed: MySQL commits each DDL statement on its own.-- +ptah no_transactionALTER TABLE deliveries ADD COLUMN status VARCHAR(16); UPDATE deliveries SET status = 'open'; ALTER TABLE deliveries ADD CONSTRAINT deliveries_status_known CHECK (statuz IN ('open', 'closed')); # Point the resource at release 2, and approve the plan it publishes.$ APPLY=OnApproval demo/bin/lab manifest deliveries "$(cat demo/.lab/deliveries-2-digest)" | kubectl apply -f -ptahmigration.operator.ptah.run/deliveries configured # The plan holds version 2 alone. Approve it by name, UID and fingerprint.$ kubectl -n "$NAMESPACE" get ptahmigration deliveries -o json > demo/.lab/deliveries.jsonkubectl -n "$NAMESPACE" get ptahmigrationplan \ "$(jq -r .status.plan.name demo/.lab/deliveries.json)" -o json > demo/.lab/deliveries-plan.jsonjq -r '.spec.migrations[] | "planned: \(.version) \(.description)"' demo/.lab/deliveries-plan.jsonjq -n --slurpfile migration demo/.lab/deliveries.json --slurpfile plan demo/.lab/deliveries-plan.json ' { apiVersion: "operator.ptah.run/v1alpha1", kind: "PtahMigrationApproval", metadata: {name: "deliveries-release-2"}, spec: { migrationRef: {name: $migration[0].metadata.name, uid: $migration[0].metadata.uid}, planRef: {name: $plan[0].metadata.name, uid: $plan[0].metadata.uid}, planFingerprint: $plan[0].spec.fingerprint } }' | kubectl -n "$NAMESPACE" create -f -planned: 2 Add Statusptahmigrationapproval.operator.ptah.run/deliveries-release-2 created # The run stopped partway. The resource records it and will not run anything until it is settled.$ kubectl -n "$NAMESPACE" get ptahmigration deliveries -o json \ | jq -r '"last run: \(.status.lastRun.outcome)", "unresolved: \(.status.unresolvedRun.outcome // "none")"'last run: Partialunresolved: Partial # First, what committed. The column exists and every row was filled.$ kubectl -n "$NAMESPACE" exec deploy/demo-mysql-client -- mysql demo -e \ "SELECT id, reference, status FROM deliveries"id reference status1 D-100 open2 D-200 open3 D-300 open # The revision table agrees. Version 2 stopped after two of its three statements.$ kubectl -n "$NAMESPACE" exec deploy/demo-mysql-client -- mysql demo -N -e \ "SELECT CONCAT('version ', version, ': ', applied, ' of ', total, ' statements, ', state) FROM schema_migrations ORDER BY version"version 1: 1 of 1 statements, appliedversion 2: 2 of 3 statements, failed # This database is disposable, so undo both statements. A real one needs its changes judged first.$ kubectl -n "$NAMESPACE" exec deploy/demo-mysql-client -- mysql demo -e \ "ALTER TABLE deliveries DROP COLUMN status; DELETE FROM schema_migrations WHERE version = 2" # Point the resource at release 1, which ends before the migration that failed.$ APPLY=OnApproval demo/bin/lab manifest deliveries "$(cat demo/.lab/deliveries-1-digest)" | kubectl apply -f -ptahmigration.operator.ptah.run/deliveries configured # Its next reading finds nothing to apply, and that reading is what clears the record.$ kubectl -n "$NAMESPACE" get ptahmigration deliveries -o json \ | jq -r '"unresolved: \(.status.unresolvedRun.outcome // "none")"'unresolved: none # Release 3 carries the corrected migration. Point the resource at it.$ APPLY=OnApproval demo/bin/lab manifest deliveries "$(cat demo/.lab/deliveries-3-digest)" | kubectl apply -f -ptahmigration.operator.ptah.run/deliveries configured # Its plan holds the corrected version 2. Approve it the same way.$ kubectl -n "$NAMESPACE" get ptahmigration deliveries -o json > demo/.lab/deliveries.jsonkubectl -n "$NAMESPACE" get ptahmigrationplan \ "$(jq -r .status.plan.name demo/.lab/deliveries.json)" -o json > demo/.lab/deliveries-plan.jsonjq -n --slurpfile migration demo/.lab/deliveries.json --slurpfile plan demo/.lab/deliveries-plan.json ' { apiVersion: "operator.ptah.run/v1alpha1", kind: "PtahMigrationApproval", metadata: {name: "deliveries-release-3"}, spec: { migrationRef: {name: $migration[0].metadata.name, uid: $migration[0].metadata.uid}, planRef: {name: $plan[0].metadata.name, uid: $plan[0].metadata.uid}, planFingerprint: $plan[0].spec.fingerprint } }' | kubectl -n "$NAMESPACE" create -f -ptahmigrationapproval.operator.ptah.run/deliveries-release-3 created # All three statements ran this time, and the three deliveries are still there.$ kubectl -n "$NAMESPACE" exec deploy/demo-mysql-client -- mysql demo -N -e \ "SELECT CONCAT('version ', version, ': ', applied, ' of ', total, ' statements, ', state) FROM schema_migrations ORDER BY version; SELECT CONCAT(id, ' ', reference, ' ', status) FROM deliveries ORDER BY id"version 1: 1 of 1 statements, appliedversion 2: 3 of 3 statements, applied1 D-100 open2 D-200 open3 D-300 open
What was checked
Every step states what has to hold before its output may be published. A condition is read as its type, status and reason on the live object, never as a phrase in a message.
| Step | Claim | Held |
|---|---|---|
| 1 | exits 0, prints "created" | yes |
| 2 | ptahmigration/deliveries: Ready=True (HistoryMatched) | yes |
| 2 | exits 0, prints "D-100", prints "D-200", prints "D-300" | yes |
| 3 | exits 0, prints "ADD COLUMN status", prints "UPDATE deliveries", prints "CHECK (statuz" | yes |
| 4 | exits 0, prints "configured" | yes |
| 5 | ptahmigration/deliveries: ApprovalRequired=True (AwaitingApproval) | yes |
| 5 | exits 0, prints "planned: 2 Add Status", prints "created" | yes |
| 6 | exits 0, prints "last run: Partial", prints "unresolved: Partial" | yes |
| 6 | ptahmigration/deliveries: Blocked=True (ApplyOutcomeUnknown) | yes |
| 7 | exits 0, prints "status", prints "open", prints "D-100", prints "D-300" | yes |
| 8 | exits 0, prints "version 1: 1 of 1 statements", prints "version 2: 2 of 3 statements" | yes |
| 9 | exits 0 | yes |
| 10 | exits 0, prints "configured" | yes |
| 11 | ptahmigration/deliveries: Ready=True (HistoryMatched) | yes |
| 11 | exits 0, prints "unresolved: none" | yes |
| 12 | exits 0, prints "configured" | yes |
| 13 | ptahmigration/deliveries: ApprovalRequired=True (AwaitingApproval) | yes |
| 13 | exits 0, prints "created" | yes |
| 14 | ptahmigration/deliveries: Ready=True (HistoryMatched) | yes |
| 14 | exits 0, prints "version 2: 3 of 3 statements", prints "1 D-100 open", prints "2 D-200 open", prints "3 D-300 open" | yes |
| 14 | ptahmigration/deliveries: status.lastRun.outcome=Applied | yes |
What it ran against
- Kubernetes1.37.0
- Operatorc08fc73ebbae
- Ptahf6e562c
- Executorsha256:137662e6db45
- Recorded2026-09-28
- Scenarios atc08fc73ebbae
What the commands read
A session is repeated, not admired, so every name in it is one your own environment can carry. There is nothing else: no helper of ours stands between a command and the cluster, and the recorder refuses a step that reads a name this list does not hold. The commands arekubectl, ptah and kubectl ptah, each of which you install once; the last one is the operator’s own read-only client.
- KUBECONFIGthe cluster the commands run against
- NAMESPACEthe namespace the schema and its database credentials live in
- OPERATOR_NAMESPACEwhere the chart installed the operator
- CONTROLLERthe controller Deployment’s name, from that release
- REGISTRY_IN_CLUSTERthe registry address a Pod in the cluster resolves
- PTAH_OCI_REGISTRYthe registry address the push goes to
- PTAH_OCI_USERNAMEthe account that push uses
- PTAH_OCI_PASSWORDits password, which no command prints
A session also uses demo/bin/lab in two places, and both are the lab handing over what it generated rather than doing the demonstration’s work:lab credentials prints the three registry values above, and lab manifestfills the published template with values the step states. Rundemo/acceptance/reproduce.sh to watch the same scenario repeated from a directory where neither exists, by an account that may not create a Job, with the plan approved by a second account that may create neither a schema nor a Job.
The scenario is demo/scenarios/mysql-partial-failure.yaml, and make demo replays the whole set against a lab of your own. All recorded runs.