PtahRecover a MySQL migration that stopped halfway
DDL that committed, a run nobody can retry, and a repair decided by what the database holds.
What it shows. How to find out what a partial MySQL migration left behind before choosing how to repair it.
Where it starts. The lab's MySQL, empty, with its URL in the Secret demo-mysql-database. Three releases of one sequence sit in demo/mysql: 1 creates the table, 2 adds a migration whose last statement fails, and 3 carries it corrected. Each becomes a ConfigMap, and a manifest per release selects its files.
A replay of a verified run, not a live cluster. 25 checks held while it was recorded. Play types it out in the frame below; the transcript is the same session, already there.
# Each release becomes a ConfigMap. The resource reads release 1; every run waits for a person.$ for release in 1 2 3; do kubectl -n "$NAMESPACE" create configmap "deliveries-release-$release" --from-file="demo/mysql/release-$release"donekubectl -n "$NAMESPACE" apply -f demo/manifests/mysql-partial-failure/deliveries-release-1.yamlconfigmap/deliveries-release-1 createdconfigmap/deliveries-release-2 createdconfigmap/deliveries-release-3 createdptahmigration.operator.ptah.run/deliveries created # The operator reads the history and plans release 1.$ kubectl ptah wait migration/deliveries -n "$NAMESPACE" --for approvalptahmigration/deliveries in namespace demo reached approval. Phase: AwaitingApproval Generation: 1, observed by the operator Database: last read at <time> Plan: ptah-mplan-764624aaaaeac40e5531dfa1kubectl ptah approve migration/deliveries -n demo reviews it. # Approve it, and wait for the table.$ kubectl ptah approve migration/deliveries -n "$NAMESPACE" --name deliveries-release-1 --waitPtahMigration: demo/deliveries, UID <uid>Plan: ptah-mplan-764624aaaaeac40e5531dfa1, UID <uid>Fingerprint: sha256:764624aaaaeac40e5531dfa15e45fde40bf853433aec771efb7681a7b85e06d7 From version 0, in this order: 1 Create Deliveries checksum 82d716605f9c339c1c58a6655b77a5e05833e0bc98d79d7862c1700150ce1b10 A migration plan records versions and checksums. Its SQL is read from thesource the plan binds, and it is not shown here: Source: ConfigMap Revision: <uid> Content: sha256:079a0e40ba36056e6cbb18540f50d266042296c22166f6f0030b39d932cfa61a Approve plan ptah-mplan-764624aaaaeac40e5531dfa1 of migration/deliveries in demo? Type yes to approve: yesApproved: PtahMigrationApproval demo/deliveries-release-1, recorded by admission as kubernetes-admin at <time>.It binds plan ptah-mplan-764624aaaaeac40e5531dfa1, fingerprint sha256:764624aaaaeac40e5531dfa15e45fde40bf853433aec771efb7681a7b85e06d7.Waiting up to 5m0s for ptahmigration/deliveries in namespace demo to reach applied.ptahmigration/deliveries in namespace demo reached applied. Phase: InSync Generation: 1, observed by the operator Database: last read at <time> Last run: Applied, Job ptah-m-apply-deliveries-08d5f2ef5f3fab0bkubectl ptah migration deliveries -n demo shows where it stands. # Three deliveries arrive. The next release has to keep them.$ kubectl -n "$NAMESPACE" exec deploy/demo-mysql-client -- mysql demo -e \ "INSERT INTO deliveries VALUES (1, 'D-100'), (2, 'D-200'), (3, 'D-300'); SELECT * FROM deliveries"id reference1 D-1002 D-2003 D-300 # Release 2 adds a status, fills it, then constrains it, naming a column that is not there.$ cat demo/mysql/release-2/0000000002_add_status.up.sql-- Adds a status, fills it, and then constrains it. The constraint names a-- column that does not exist, so the third statement fails after the first two-- have committed: MySQL commits each DDL statement on its own.-- +ptah no_transactionALTER TABLE deliveries ADD COLUMN status VARCHAR(16); UPDATE deliveries SET status = 'open'; ALTER TABLE deliveries ADD CONSTRAINT deliveries_status_known CHECK (statuz IN ('open', 'closed')); # Point the resource at release 2. The manifest names the next ConfigMap and its new files.$ diff -u demo/manifests/mysql-partial-failure/deliveries-release-1.yaml demo/manifests/mysql-partial-failure/deliveries-release-2.yamlkubectl -n "$NAMESPACE" apply -f demo/manifests/mysql-partial-failure/deliveries-release-2.yaml--- demo/manifests/mysql-partial-failure/deliveries-release-1.yaml 2026-11-06 15:43:50.129445411 +0000+++ demo/manifests/mysql-partial-failure/deliveries-release-2.yaml 2026-11-06 15:43:50.129549857 +0000@@ -20,12 +20,16 @@ key: url artifact: configMap:- name: deliveries-release-1+ name: deliveries-release-2 items: - key: 0000000001_create_deliveries.up.sql path: 0000000001_create_deliveries.up.sql - key: 0000000001_create_deliveries.down.sql path: 0000000001_create_deliveries.down.sql+ - key: 0000000002_add_status.up.sql+ path: 0000000002_add_status.up.sql+ - key: 0000000002_add_status.down.sql+ path: 0000000002_add_status.down.sql policy: # Every run waits for a person. apply: OnApprovalptahmigration.operator.ptah.run/deliveries configured # The plan holds version 2 alone.$ kubectl ptah wait migration/deliveries -n "$NAMESPACE" --for approvalptahmigration/deliveries in namespace demo reached approval. Phase: AwaitingApproval Generation: 2, observed by the operator Database: last read at <time> Plan: ptah-mplan-326d290ba57156a43719f067kubectl ptah approve migration/deliveries -n demo reviews it. # Approve it. The run stops partway, and the wait says it will not end applied.$ kubectl ptah approve migration/deliveries -n "$NAMESPACE" --name deliveries-release-2 --waitPtahMigration: demo/deliveries, UID <uid>Plan: ptah-mplan-326d290ba57156a43719f067, UID <uid>Fingerprint: sha256:326d290ba57156a43719f067f927c9aac56c077b0ae9157a7f00ba98e8627b71 From version 1, in this order: 2 Add Status checksum 5dee45a150440c6d26354f1467f448fbcf2a02116cd093828f3497a1b192d46b, transaction none A migration plan records versions and checksums. Its SQL is read from thesource the plan binds, and it is not shown here: Source: ConfigMap Revision: <uid> Content: sha256:7b4b9833ad7bd5a6472fbc28f3ed750889cb49aa4c20ddd7e03986d4fa4549f5 Approve plan ptah-mplan-326d290ba57156a43719f067 of migration/deliveries in demo? Type yes to approve: yesApproved: PtahMigrationApproval demo/deliveries-release-2, recorded by admission as kubernetes-admin at <time>.It binds plan ptah-mplan-326d290ba57156a43719f067, fingerprint sha256:326d290ba57156a43719f067f927c9aac56c077b0ae9157a7f00ba98e8627b71.Waiting up to 5m0s for ptahmigration/deliveries in namespace demo to reach applied.kubectl-ptah: ptahmigration/deliveries in namespace demo will not reach applied: OutcomeUnknown. Phase: Blocked Why: the run of plan ptah-mplan-326d290ba57156a43719f067 (Job ptah-m-apply-deliveries-d39d4ab836ba46f0) ended Partial: its effect on the database is not established, and nothing runs until a reading or a person accounts for itkubectl ptah migration deliveries -n demo shows where it stands. # The last run ended Partial, and an unresolved run holds the resource. Two different records.$ kubectl ptah migration deliveries -n "$NAMESPACE"Migration: demo/deliveriesPhase: BlockedReady: False, ApplyOutcomeUnknown: The database has to be read by a person before anything else runsBlocked: True, ApplyOutcomeUnknown: Migration 2 (Add Status) committed 2 of its 3 statements, then failed with SQLSTATE 42S22 (error 1054); retrying the file would run them twicePolicy: apply OnApprovalSource: ConfigMap deliveries-release-2Snapshot: ptah-src-a3da8502bfbe74f678b2bcd8Content: sha256:7b4b9833ad7bd5a6472fbc28f3ed750889cb49aa4c20ddd7e03986d4fa4549f5History read: <time>Current version: 1Applied: 1Pending: 1 Last run: Partial, started <time>, finished <time> Job: ptah-m-apply-deliveries-d39d4ab836ba46f0, UID <uid> Stopped at: version 2, statement 3 of 3, SQLSTATE 42S22, error 1054 Message: Migration 2 (Add Status) committed 2 of its 3 statements, then failed with SQLSTATE 42S22 (error 1054); retrying the file would run them twice Unresolved run: Partial, recorded <time> Operation: sha256:37caeeb7fd1a7327de00243b63b4d6c7bf97b80723c2b4fd5619abeb87b9337a Job: ptah-m-apply-deliveries-d39d4ab836ba46f0, UID <uid> Plan: ptah-mplan-326d290ba57156a43719f067, UID <uid> Database: sha256:1a2e9094d366828c3ce1ca286ad6d9931ce4cc3b5d020554d68fb53feb9a02c5 (target identity digest) While this record stands no plan is published and no Apply is dispatched; resolve, verify and the history read go on. The database is the one the run reported reaching or, where it reported nothing, the one it was dispatched against: where to start looking, not proof of where the run went. A reader who inspects the wrong database finds nothing wrong and settles a record that was telling the truth. Establish what the run did there first. A history read of that database that finds nothing of the artifact left to apply clears the record by itself. If the database is accounted for and still has work pending, settle it with a PtahMigrationRunAcknowledgment naming migration UID <uid> and the operation above. The operations guide has the procedure, under "A migration run nobody accounted for". No plan is published. Approvals, newest first: deliveries-release-2 approved by kubernetes-admin at <time> plan ptah-mplan-326d290ba57156a43719f067, not the current plan TYPE STATUS REASON SINCE MESSAGE Consumed True DispatchCommitted <time> The approved plan was dispatched deliveries-release-1 approved by kubernetes-admin at <time> plan ptah-mplan-764624aaaaeac40e5531dfa1, not the current plan TYPE STATUS REASON SINCE MESSAGE Consumed True DispatchCommitted <time> The approved plan was dispatched Run acknowledgments: none Conditions: TYPE STATUS REASON SINCE MESSAGE Progressing False ApplyOutcomeUnknown <time> The run stopped and may not be retried Blocked True ApplyOutcomeUnknown <time> Migration 2 (Add Status) committed 2 of its 3 statements, then failed with SQLSTATE 42S22 (error 1054); retrying the file would run them twice Ready False ApplyOutcomeUnknown <time> The database has to be read by a person before anything else runs DatabaseReachable True Observed <time> The Apply opened the database ApprovalRequired False Satisfied <time> The plan's approval requirements are satisfied kubectl events -n demo --for ptahmigration/deliveries prints its Events. # First, what committed. The column exists and every row was filled.$ kubectl -n "$NAMESPACE" exec deploy/demo-mysql-client -- mysql demo -e "SELECT id, reference, status FROM deliveries ORDER BY id"id reference status1 D-100 open2 D-200 open3 D-300 open # The revision table agrees. Version 2 stopped after two of its three statements.$ kubectl -n "$NAMESPACE" exec deploy/demo-mysql-client -- mysql demo -e \ "SELECT version, applied, total, state FROM schema_migrations ORDER BY version"version applied total state1 1 1 applied2 2 3 failed # Disposable lab only: undo both statements by hand. A real database needs them judged first.$ kubectl -n "$NAMESPACE" exec deploy/demo-mysql-client -- mysql demo -e \ "ALTER TABLE deliveries DROP COLUMN status; DELETE FROM schema_migrations WHERE version = 2" # Point the resource at release 1, which ends before the migration that failed.$ kubectl -n "$NAMESPACE" apply -f demo/manifests/mysql-partial-failure/deliveries-release-1.yamlptahmigration.operator.ptah.run/deliveries configured # Its next reading of the history finds nothing to apply, and that reading clears the record.$ kubectl ptah wait migration/deliveries -n "$NAMESPACE" --for in-syncptahmigration/deliveries in namespace demo reached in-sync. Phase: InSync Generation: 3, observed by the operator Database: last read at <time>kubectl ptah migration deliveries -n demo shows where it stands. # The hold is gone, and the resource says what settled it.$ kubectl ptah migration deliveries -n "$NAMESPACE"Migration: demo/deliveriesPhase: InSyncReady: True, HistoryMatched: The database has every migration this artifact carriesPolicy: apply OnApprovalSource: ConfigMap deliveries-release-1Snapshot: ptah-src-8419fbac2e7266e67b6cc2bfContent: sha256:079a0e40ba36056e6cbb18540f50d266042296c22166f6f0030b39d932cfa61aHistory read: <time>Current version: 1Applied: 1Pending: 0 Last run: Partial, started <time>, finished <time> Job: ptah-m-apply-deliveries-d39d4ab836ba46f0, UID <uid> Stopped at: version 2, statement 3 of 3, SQLSTATE 42S22, error 1054 Message: Migration 2 (Add Status) committed 2 of its 3 statements, then failed with SQLSTATE 42S22 (error 1054); retrying the file would run them twice Settled run: Partial run sha256:37caeeb7fd1a7327de00243b63b4d6c7bf97b80723c2b4fd5619abeb87b9337a, by a history read at <time> No plan is published. Approvals, newest first: deliveries-release-2 approved by kubernetes-admin at <time> plan ptah-mplan-326d290ba57156a43719f067, not the current plan TYPE STATUS REASON SINCE MESSAGE Consumed True DispatchCommitted <time> The approved plan was dispatched deliveries-release-1 approved by kubernetes-admin at <time> plan ptah-mplan-764624aaaaeac40e5531dfa1, not the current plan TYPE STATUS REASON SINCE MESSAGE Consumed True DispatchCommitted <time> The approved plan was dispatched Run acknowledgments: none Conditions: TYPE STATUS REASON SINCE MESSAGE Progressing False HistoryMatched <time> Nothing is pending Blocked False HistoryMatched <time> The history continues this artifact Ready True HistoryMatched <time> The database has every migration this artifact carries DatabaseReachable True Observed <time> The migration history was read ApprovalRequired False Satisfied <time> The plan's approval requirements are satisfied (written for generation 2) kubectl events -n demo --for ptahmigration/deliveries prints its Events. # Release 3 carries the corrected migration. Point the resource at it.$ diff -u demo/manifests/mysql-partial-failure/deliveries-release-2.yaml demo/manifests/mysql-partial-failure/deliveries-release-3.yamlkubectl -n "$NAMESPACE" apply -f demo/manifests/mysql-partial-failure/deliveries-release-3.yaml--- demo/manifests/mysql-partial-failure/deliveries-release-2.yaml 2026-11-06 15:43:50.129549857 +0000+++ demo/manifests/mysql-partial-failure/deliveries-release-3.yaml 2026-11-06 15:43:50.129549857 +0000@@ -20,7 +20,7 @@ key: url artifact: configMap:- name: deliveries-release-2+ name: deliveries-release-3 items: - key: 0000000001_create_deliveries.up.sql path: 0000000001_create_deliveries.up.sqlptahmigration.operator.ptah.run/deliveries configured # Its plan holds the corrected version 2.$ kubectl ptah wait migration/deliveries -n "$NAMESPACE" --for approvalptahmigration/deliveries in namespace demo reached approval. Phase: AwaitingApproval Generation: 4, observed by the operator Database: last read at <time> Plan: ptah-mplan-a878adf5aaa37100b7d3fcd8kubectl ptah approve migration/deliveries -n demo reviews it. # Approve it the same way.$ kubectl ptah approve migration/deliveries -n "$NAMESPACE" --name deliveries-release-3 --waitPtahMigration: demo/deliveries, UID <uid>Plan: ptah-mplan-a878adf5aaa37100b7d3fcd8, UID <uid>Fingerprint: sha256:a878adf5aaa37100b7d3fcd8d06d9c73dd5cce7ba36a7577fdf98db85d5581d1 From version 1, in this order: 2 Add Status checksum 0f213750642209f38a779a6ef7bac1c8997f339b51db4e59b013c12d5b59e5cb, transaction none A migration plan records versions and checksums. Its SQL is read from thesource the plan binds, and it is not shown here: Source: ConfigMap Revision: <uid> Content: sha256:41016098228262638ebd6b8dedbab65b55a7a979d0bb8c0a30cf218bc686c2e3 Approve plan ptah-mplan-a878adf5aaa37100b7d3fcd8 of migration/deliveries in demo? Type yes to approve: yesApproved: PtahMigrationApproval demo/deliveries-release-3, recorded by admission as kubernetes-admin at <time>.It binds plan ptah-mplan-a878adf5aaa37100b7d3fcd8, fingerprint sha256:a878adf5aaa37100b7d3fcd8d06d9c73dd5cce7ba36a7577fdf98db85d5581d1.Waiting up to 5m0s for ptahmigration/deliveries in namespace demo to reach applied.ptahmigration/deliveries in namespace demo reached applied. Phase: InSync Generation: 4, observed by the operator Database: last read at <time> Last run: Applied, Job ptah-m-apply-deliveries-57fdd3de247e4f4dkubectl ptah migration deliveries -n demo shows where it stands. # All three statements ran this time, and the three deliveries are still there.$ kubectl -n "$NAMESPACE" exec deploy/demo-mysql-client -- mysql demo -e \ "SELECT version, applied, total, state FROM schema_migrations ORDER BY version; SELECT id, reference, status FROM deliveries ORDER BY id"version applied total state1 1 1 applied2 3 3 appliedid reference status1 D-100 open2 D-200 open3 D-300 open
What was checked
Every step states what has to hold before its output may be published. A condition is read as its type, status and reason on the live object, never as a phrase in a message.
| Step | Claim | Held |
|---|---|---|
| 1 | exits 0, prints "configmap/deliveries-release-1 created", prints "configmap/deliveries-release-2 created", prints "configmap/deliveries-release-3 created", prints "deliveries created" | yes |
| 2 | exits 0, prints "reached approval" | yes |
| 2 | ptahmigration/deliveries: ApprovalRequired=True (AwaitingApproval) | yes |
| 3 | exits 0, prints "Create Deliveries", prints "Type yes to approve: yes", prints "Approved: PtahMigrationApproval", prints "reached applied" | yes |
| 4 | exits 0, prints "1\tD-100", prints "2\tD-200", prints "3\tD-300" | yes |
| 5 | exits 0, prints "ADD COLUMN status", prints "UPDATE deliveries", prints "CHECK (statuz" | yes |
| 6 | exits 0, prints "+ name: deliveries-release-2", prints "deliveries configured" | yes |
| 7 | exits 0, prints "reached approval" | yes |
| 7 | ptahmigration/deliveries: ApprovalRequired=True (AwaitingApproval) | yes |
| 8 | exits 8, prints "Add Status", prints "Type yes to approve: yes", prints "Approved: PtahMigrationApproval", prints "will not reach applied", prints "Partial" | yes |
| 9 | exits 0, prints "Last run: Partial", prints "Unresolved run: Partial", prints "no Apply is dispatched" | yes |
| 9 | ptahmigration/deliveries: Blocked=True (ApplyOutcomeUnknown) | yes |
| 10 | exits 0, prints "1\tD-100\topen", prints "3\tD-300\topen" | yes |
| 11 | exits 0, prints "version\tapplied\ttotal\tstate", prints "1\t1\t1\tapplied", prints "2\t2\t3\t" | yes |
| 12 | exits 0 | yes |
| 13 | exits 0, prints "deliveries configured" | yes |
| 14 | exits 0, prints "reached in-sync" | yes |
| 14 | ptahmigration/deliveries: Ready=True (HistoryMatched) | yes |
| 15 | exits 0, prints "Settled run:", prints "by a history read", prints "Last run: Partial", never prints "Unresolved run:" | yes |
| 16 | exits 0, prints "+ name: deliveries-release-3", prints "deliveries configured" | yes |
| 17 | exits 0, prints "reached approval" | yes |
| 17 | ptahmigration/deliveries: ApprovalRequired=True (AwaitingApproval) | yes |
| 18 | exits 0, prints "Add Status", prints "Type yes to approve: yes", prints "Approved: PtahMigrationApproval", prints "reached applied" | yes |
| 19 | exits 0, prints "2\t3\t3\tapplied", prints "1\tD-100\topen", prints "2\tD-200\topen", prints "3\tD-300\topen" | yes |
| 19 | ptahmigration/deliveries: status.lastRun.outcome=Applied | yes |
What it ran against
- Kubernetes1.37.0
- Operatore41a04412bb1
- Ptahv0.18.0
- Executorsha256:9f8797986124
- Recorded2026-11-06
- Scenarios ate41a04412bb1
What the commands read
A session is repeated, not admired, so every name in it is one your own environment can carry. There is nothing else: no helper of ours stands between a command and the cluster, and the recorder refuses a step that reads a name this list does not hold. The commands arekubectl, ptah and kubectl ptah, each of which you install once; the last one is the operator’s own client. They run from the root of a clone of this repository, which holds the files they name: the SQL under demo/schemas, demo/migrations,demo/migrations-checkpoint and demo/mysql, the reference data underdemo/reference, and the manifests under demo/manifests.
- KUBECONFIGthe cluster the commands run against
- NAMESPACEthe namespace the resources and their database credentials live in
- OPERATOR_NAMESPACEwhere the chart installed the operator
- CONTROLLERthe controller Deployment’s name, from that release
- EXECUTOR_IMAGEthe executor image the chart runs, its
executorImagevalue - REGISTRY_IN_CLUSTERthe registry address a Pod in the cluster resolves
- PTAH_OCI_REGISTRYthe registry address the push goes to
- PTAH_OCI_USERNAMEthe account that push uses, read by
ptahitself - PTAH_OCI_PASSWORDits password, which no command prints
The last two are how a CI job hands ptah a registry credential. On a workstation,ptah oci login stores it once instead, reading the password from a prompt or from standard input rather than from the command line.
The registry has two addresses, the way a registry in an ordinary deployment does: a push goes to PTAH_OCI_REGISTRY, and the resource names REGISTRY_IN_CLUSTER with the same repository and digest. The registry Secret a resource names says which address its credential is for, and the operator sends it to no other.
The lab itself appears in one command, demo/bin/lab versions in Stand up the lab, which says what the lab is made of. What a run starts from is its preparation, which is not recorded; Where it starts above says what that is and which run ends there. Rundemo/acceptance/reproduce.sh to watch Approve a plan repeated from a directory where the lab’s scripts do not exist, by an account that may not create a Job, with the plan approved by a second account that may create neither a schema nor a Job, and then the same schema published with ptah schema push and moved to its next revision withkubectl ptah set-source.
The same changes, as edits
Each kubectl ptah command that changes a resource changes one field of it, so the same change is an ordinary edit, with kubectl edit ptahschema storefront -n "$NAMESPACE"or in a manifest you apply; the runs that keep their resources as files underdemo/manifests make their changes that way. What the commands add is checking: set-source takes only a reference pinned by digest, changes nothing while an Apply may be running, and changes nothing another writer changed since it read the resource.
| Command | The line it changes |
|---|---|
kubectl ptah suspend | spec.suspend: true |
kubectl ptah resume | spec.suspend: false |
kubectl ptah set-source --oci | spec.desired.ociRef of a schema, spec.artifact.ociRef of a migration |
kubectl ptah refresh | the annotation operator.ptah.run/read-requested-at, set to the time of the request |
An approval is not one of them. It is an object of its own, created once by the person who decided: kubectl ptah approve --dry-run client prints it as YAML for a repository, to be created with kubectl create, and never edited afterwards.
The scenario is demo/scenarios/mysql-partial-failure.yaml, and make demo replays the whole set against a lab of your own. All recorded runs.