Skip to content

Close the drift

Someone changes the database by hand. The operator notices and puts it back.

What it shows. What drift detection reads, and what it does about what it finds.

Where it starts. Where Change the schema ends: storefront converged on schemas/v2.sql, published as an artifact, under apply Always, reading the database every minute.

A replay of a verified run, not a live cluster. 8 checks held while it was recorded. Play types it out in the frame below; the transcript is the same session, already there.

sh · drift
# Converged, and the operator says so with a reason and the time it last read the database.$ kubectl ptah schema storefront -n "$NAMESPACE"Schema:               demo/storefrontPhase:                InSyncReady:                True, InSync: Schema is in syncPolicy:               apply Always, allowDestructive false, driftSeverity allArtifact:             oci://e2e-registry.demo.svc.cluster.local:5000/schemas/demo@sha256:a166d562143b4f650dd3ec5cc6e8a62919e6c191a352fa108fd5019f0bb46d4bVerified:             trueObserved:             <time>Drift:                noneApplied:              ptah-plan-4936145202ac9b897a444c5a at <time> No plan is published. Approvals: none Conditions:  TYPE                  STATUS  REASON           SINCE                 MESSAGE  EngineSupported       True    SupportedEngine  <time>                Database engine "PostgreSQL" is supported  Ready                 True    InSync           <time>                Schema is in sync  ReconciliationFailed  False   Succeeded        <time>                The latest operation completed  ArtifactResolved      True    DigestPinned     <time>                Desired OCI reference resolved to immutable content  ArtifactVerified      True    PolicySatisfied  <time>                Artifact type and verification policy were satisfied  DatabaseReachable     True    Observed         <time>                Database schema was observed  DriftDetected         False   ScopedConverged  <time>                The authoritative managed scope has no changes  InSync                True    ScopedConverged  <time>                A stable scoped plan proves convergence  ApprovalRequired      False   Satisfied        <time>                Current plan approval requirements are satisfied  PlanReady             False   NoChanges        <time>                No executable plan is required  Applying              False   JobCompleted     <time>                Apply Job completed; convergence observation is pending  Suspended             False   Active           <time>                Reconciliation is active kubectl events -n demo --for ptahschema/storefront prints its Events. # Fault injection: change the database behind the operator's back, the way an incident does.$ kubectl -n "$NAMESPACE" exec deploy/demo-psql -- psql -c "ALTER TABLE customers DROP COLUMN signed_up_at"ALTER TABLE # The column is gone. The declaration still asks for it.$ kubectl -n "$NAMESPACE" exec deploy/demo-psql -- psql -c "\d customers"             Table "public.customers" Column |  Type  | Collation | Nullable | Default--------+--------+-----------+----------+--------- id     | bigint |           | not null | email  | text   |           | not null |Indexes:    "customers_pkey" PRIMARY KEY, btree (id)Referenced by:    TABLE "orders" CONSTRAINT "orders_customer_id_fkey" FOREIGN KEY (customer_id) REFERENCES customers(id) # Nobody is asked. The next reading finds the drift, plans, applies, and reads again.$ kubectl ptah wait schema/storefront -n "$NAMESPACE" --for in-syncptahschema/storefront in namespace demo reached in-sync.  Phase:      InSync  Generation: 1, observed by the operator  Database:   last read at <time>kubectl ptah schema storefront -n demo shows where it stands. # The column is back.$ kubectl -n "$NAMESPACE" exec deploy/demo-psql -- psql -c "\d customers"                         Table "public.customers"    Column    |           Type           | Collation | Nullable | Default--------------+--------------------------+-----------+----------+--------- id           | bigint                   |           | not null | email        | text                     |           | not null | signed_up_at | timestamp with time zone |           |          |Indexes:    "customers_pkey" PRIMARY KEY, btree (id)Referenced by:    TABLE "orders" CONSTRAINT "orders_customer_id_fkey" FOREIGN KEY (customer_id) REFERENCES customers(id) # It closed the drift with a plan, the same way it does everything else.$ kubectl ptah plan storefront --applied -n "$NAMESPACE" -o sql-- Add/modify columns for table: customersALTER TABLE "customers" ADD COLUMN "signed_up_at" timestamptz;

What was checked

Every step states what has to hold before its output may be published. A condition is read as its type, status and reason on the live object, never as a phrase in a message.

StepClaimHeld
1exits 0, prints "Observed:", prints "Drift:", prints "ScopedConverged"yes
1ptahschema/storefront: DriftDetected=False (ScopedConverged)yes
2exits 0, prints "ALTER TABLE"yes
3exits 0, never prints "signed_up_at"yes
4exits 0, prints "reached in-sync"yes
4ptahschema/storefront: DriftDetected=False (ScopedConverged)yes
5exits 0, prints "signed_up_at"yes
6exits 0, prints "ADD COLUMN", prints "signed_up_at"yes

What it ran against

  • Kubernetes1.37.0
  • Operatore41a04412bb1
  • Ptahv0.18.0
  • Executorsha256:9f8797986124
  • Recorded2026-11-06
  • Scenarios ate41a04412bb1

What the commands read

A session is repeated, not admired, so every name in it is one your own environment can carry. There is nothing else: no helper of ours stands between a command and the cluster, and the recorder refuses a step that reads a name this list does not hold. The commands arekubectl, ptah and kubectl ptah, each of which you install once; the last one is the operator’s own client. They run from the root of a clone of this repository, which holds the files they name: the SQL under demo/schemas, demo/migrations,demo/migrations-checkpoint and demo/mysql, the reference data underdemo/reference, and the manifests under demo/manifests.

  • KUBECONFIGthe cluster the commands run against
  • NAMESPACEthe namespace the resources and their database credentials live in
  • OPERATOR_NAMESPACEwhere the chart installed the operator
  • CONTROLLERthe controller Deployment’s name, from that release
  • EXECUTOR_IMAGEthe executor image the chart runs, its executorImage value
  • REGISTRY_IN_CLUSTERthe registry address a Pod in the cluster resolves
  • PTAH_OCI_REGISTRYthe registry address the push goes to
  • PTAH_OCI_USERNAMEthe account that push uses, read by ptah itself
  • PTAH_OCI_PASSWORDits password, which no command prints

The last two are how a CI job hands ptah a registry credential. On a workstation,ptah oci login stores it once instead, reading the password from a prompt or from standard input rather than from the command line.

The registry has two addresses, the way a registry in an ordinary deployment does: a push goes to PTAH_OCI_REGISTRY, and the resource names REGISTRY_IN_CLUSTER with the same repository and digest. The registry Secret a resource names says which address its credential is for, and the operator sends it to no other.

The lab itself appears in one command, demo/bin/lab versions in Stand up the lab, which says what the lab is made of. What a run starts from is its preparation, which is not recorded; Where it starts above says what that is and which run ends there. Rundemo/acceptance/reproduce.sh to watch Approve a plan repeated from a directory where the lab’s scripts do not exist, by an account that may not create a Job, with the plan approved by a second account that may create neither a schema nor a Job, and then the same schema published with ptah schema push and moved to its next revision withkubectl ptah set-source.

The same changes, as edits

Each kubectl ptah command that changes a resource changes one field of it, so the same change is an ordinary edit, with kubectl edit ptahschema storefront -n "$NAMESPACE"or in a manifest you apply; the runs that keep their resources as files underdemo/manifests make their changes that way. What the commands add is checking: set-source takes only a reference pinned by digest, changes nothing while an Apply may be running, and changes nothing another writer changed since it read the resource.

CommandThe line it changes
kubectl ptah suspendspec.suspend: true
kubectl ptah resumespec.suspend: false
kubectl ptah set-source --ocispec.desired.ociRef of a schema, spec.artifact.ociRef of a migration
kubectl ptah refreshthe annotation operator.ptah.run/read-requested-at, set to the time of the request

An approval is not one of them. It is an object of its own, created once by the person who decided: kubectl ptah approve --dry-run client prints it as YAML for a repository, to be created with kubectl create, and never edited afterwards.

The scenario is demo/scenarios/drift.yaml, and make demo replays the whole set against a lab of your own. All recorded runs.