Events
kubectl describe ptahschema orders prints these under the status, and they are
the fastest way to see what the operator has just done.
kubectl events -n <namespace> --for ptahschema/orders prints them alone, and
--watch follows the ones that come next. They are diagnostic
text: read them to understand, and match on Conditions to decide.
Condition reasons is the interface for deciding, and
says why.
An Event is recorded on the resource a person would look at, which is not always the one the operator was working on: an approval that went stale carries the Event, because that is the object whose author has to act.
Event messages are bounded. A failure long enough to be truncated is recorded in full on the resource’s own status and in the operation’s Job.
Result reads use the transport recorded on the Job. With the default durable
delivery, the controller reads PtahResultRecord objects through the Kubernetes
API. Check API availability and the manager’s read permissions when those reads
fail. Logs and termination summaries cannot replace a missing durable receipt.
The log-loss window and summary fallback apply only to
legacy result reads.
| Reason | Type | Recorded on | What it says |
|---|---|---|---|
AdmissionSnapshotRefreshed |
Normal | PtahSchema, PtahMigration |
A claim that had not dispatched found the Pod template it would dispatch different from the one its admission snapshot recorded, usually because a new manager release took over. The claim stands, and the snapshot is resolved again before its Job is created. This happens once per claim; a template that changes again retires the claim. |
ApplyOutcomeUnknown |
Warning | PtahSchema |
An Apply ended with no proof of whether it changed the database, so a read-only observation starts to establish what happened. When the result named why the target could not be opened, the Event carries the operator’s sentence for that kind; any other error keeps the fixed text. When the run wrote something to read, the message also says where the Pod log is and for how long. What the kubelet last reported about a Pod it could not start is not in this Event: on the endings Before the Secret exists lists, ReconciliationFailed carries it. |
ApprovalAccepted |
Normal | PtahSchema |
An authenticated approval was accepted for the current immutable plan. |
ApprovalRequired |
Normal | PtahSchema |
The current immutable plan needs an approval before anything applies it. One Event is one request for a decision on one plan. A plan that asks when it is published records it on the status write that records the plan’s PlanPublished entry in status.journal with approvalRequired: true. A scheduled read clears the plan it waits on and publishes it again when the database has not changed, and records nothing. A plan already published records one later, with no journal entry beside it, each time it starts asking: its requirement comes back after something withdrew it (an approval revoked before its Apply started, a realm refusal that lifted, or a database that converged and then drifted back to that plan), or a change to spec.policy.apply makes it ask, as an edit from Never to OnApproval does. apply is not part of what a plan is computed under, so the read after that edit publishes the same plan. ptah_operator_approvals_total{outcome="required"} counts these Events, and an Event can be lost. |
ApprovalRevoked |
Warning | PtahSchema |
The recorded approval became invalid before the Apply Job was created, so no Job was created. |
ApprovalStale |
Warning | PtahSchemaApproval, PtahMigrationApproval |
This approval no longer matches the plan it was written for, the verification policy refused the artifact, or a run spent another approval of the same migration plan, and it will not be consumed. A PtahSchemaApproval an Apply already consumed is marked too, by the read after that Apply. |
ArtifactVerificationRefused |
Warning | PtahSchema, PtahMigration |
Verification refused the resolved artifact, so no plan is produced from it. The message names the requirements the policy found unmet. |
ExecutionBindingChanged |
Warning | PtahSchema, PtahMigration |
The execution binding the evidence was produced under was retired: the executor image, the Ptah version, the runner protocol or the controller-state version changed. A schema closes the approval boundary and refreshes read-only; a migration discards the operation claim. A manager release that changes none of them records no such Event. |
FailedApplyLogShortened |
Warning | PtahSchema, PtahMigration |
Admission refused the 24 hours a failed Apply’s Job was due, usually because the guard of the previous release is still installed, so the Job keeps its Pod log for five minutes. |
LeaseContinuityLost |
Warning | PtahSchema |
Database lock ownership was not continuous, so the result it covered is discarded and the proof restarts. |
MigrationJobCleanupDeferred |
Warning | PtahMigration |
A Job’s cleanup TTL could not be scheduled, and the message says why. The Job stays until it is deleted by hand or with its owner, and holds a place in a count/jobs.batch quota until then. For a Job still running when its run was recorded Unknown, the TTL is asked for once the Job stops, and this Event comes then if admission refuses it. Admission admits the TTL only on the Job a claim names by UID, so a claim that never recorded the UID, as when the manager stopped between creating the Job and recording it, records it before the TTL is asked for, where the Job is the one the claim fixes. A Job under the reserved name that is not, such as one under another execution epoch, is never recorded, and this Event names it, with the Job is not the one the claim fixes, when the claim goes. A read-only Job its claim could not confirm, as after a release that changed the Job, gets its TTL once it stops, before the claim runs again. This Event names one admission would refuse, such as one under another execution epoch or one whose UID its claim never recorded, when the claim runs again without it. It also names one that has not finished by its own deadline and the termination grace its Pods are given, with did not stop by its own deadline, since the claim stops waiting for it then. |
MigrationRunFinished |
Normal or Warning | PtahMigration |
An Apply finished and the message names what it did. The type follows the outcome: Applied and UpToDate are Normal, and Failed, Partial and Unknown are Warnings, and those say where the Pod log is, unless the run could not open its target. |
MigrationRunUncertain |
Warning | PtahMigration |
A migration run ended without establishing what it committed, and the resource records that until something resolves it. The message says where the run’s Pod log is, unless its result names nothing there to read or an init container ended it. On the endings Before the Secret exists lists, among them a Job deleted, replaced or left without its owner, a changed execution component, a database Lease that changed hands and a deleted resource, it also carries what the kubelet last reported about a Pod it could not start. For a deleted resource this Event is where those words stay. |
OperationCompleted |
Normal | PtahSchema |
An operation finished and its result was accepted. |
OperationFailed |
Warning | PtahSchema, PtahMigration |
An operation failed. The message carries the failure; for a desired file Ptah could not read, Ptah’s reason, and for a database the operation could not open, its kind and the Secret the URL came from, as A database the operation could not open lists. For a schema whose target moved it says so and names ReconciliationFailed, which holds the command, since an Event keeps too little of it. |
OperationRetried |
Warning | PtahMigration |
An attempt failed and another will follow. The message names the attempt number and the failure, as OperationFailed does. |
OperationStarted |
Normal | PtahSchema, PtahMigration |
A Job was created for the named operation. |
PlanPublicationRefused |
Warning | PtahSchema |
The API server refused to store the plan a Plan operation computed, and the message carries the refusal: an admission policy or webhook, a quota, RBAC, or the object’s size. The Plan runs again after the failure interval, so a policy or quota that stops refusing lets the next plan through. |
PlanStale |
Warning | PtahSchema |
Another plan replaced a plan a person could approve, or had approved, before it ran. A plan a realm refusal held back counts as well. The message names both. An approval of the old plan no longer counts, so read the new one with kubectl ptah plan and approve that if it still needs an approval. A scheduled read that publishes the same plan records nothing, and neither does an Apply that ran its plan or a database that converged, which leaves nothing in the plan’s place. A plan that apply policy Always or Never would never wait on, or that allowDestructive blocks, records nothing either: nobody could have approved it. This Event fires once for each replacement and can be lost, and the PlanPublished entry in status.journal names the replaced plan as replaced. |
PlanTargetChanged |
Warning | PtahSchema |
The database target changed between the read and the plan computed from it, so that plan was not published and the database is read again. |
PodAdmissionRefused |
Warning | PtahSchema, PtahMigration |
The Job controller could not create the operation Pod because the API server refused it, and the message carries the refusal: a policy the namespace runs, a mutating admission whose change the operator’s own Pod webhook refused, a quota. The Job keeps its deadline and the Job controller keeps trying; declare what the Pod has to carry in spec.execution.podMetadata, or change the policy. Execution says what a Pod may carry. |
PodConfigurationError |
Warning | PtahSchema, PtahMigration |
The operation Pod exists and the kubelet cannot start its container, and the message carries the kubelet’s words: usually a Secret or a key that does not exist yet. The kubelet keeps trying and the Job keeps its deadline, so a Secret that arrives lets the operation go on. Before the Secret exists has the detail. |
ProofSettled |
Normal | PtahSchema |
A person wrote the value the stop printed in the operator.ptah.run/settled-apply annotation, so the operator gave up the post-Apply proof and reads the database again under the URL the Secret holds. No Apply is recorded, and under apply: Always the next plan applies with no approval unless it is destructive or changes privileges. A schema whose target moved says what that statement means. |
ProtectedTableRefused |
Warning | PtahSchema |
A plan would change a table spec.policy.protectedTables fences off, so no plan is published. |
ReadRequested |
Normal | PtahSchema, PtahMigration |
A read request was answered, and the read the interval would have started starts now instead. On a PtahSchema waiting on a plan, it starts after the Apply an already recorded approval authorizes, and the message says so. |
ReadRequestInvalid |
Warning | PtahSchema, PtahMigration |
The operator.ptah.run/read-requested-at annotation holds a value that is not an RFC 3339 instant, which the message names. It is recorded as answered in status.lastReadRequest and starts nothing, so this is sent once, by a read that keeps failing as well as by a resource at rest, and never again for that value. It can be lost, and beside a failing read it usually is: every failed attempt sends a Warning too, so after about a quarter of an hour of failures at the default 30-second retry interval the recorder drops it. status.lastReadRequest is the record: a value there that is not an RFC 3339 instant was answered and started no read. Set an instant to ask for a read. Reading again on request has the rules. |
ReadRestarted |
Normal | PtahSchema, PtahMigration |
A read-only operation that kept failing gave way to a new Resolve, because its source changed, a read was requested, or its interval passed. The message says which. A read that keeps failing has the rules. |
ResultCredentialFailed |
Warning | PtahSchema, PtahMigration |
Issuing the operation-bound delivery credential failed and will be retried. Check receiver trust readiness, the enrollment policy, and admission refusals. The runner waits for its credential before executing. |
ResultReadFailed |
Warning | PtahSchema, PtahMigration |
Reading an operation’s result failed and will be retried. The message identifies the failed read; check the Job’s result transport before investigating storage or container logs. |
ResultReadTimedOut |
Warning | PtahSchema, PtahMigration |
Reading an operation’s result exceeded its time bound and will be retried. This can affect either durable result storage or legacy container-log reads. |
ResultRetirementQuotaExceeded |
Warning | PtahSchema, PtahMigration |
A ResourceQuota on result records refused the retired record that starts an operation’s retention window, and the message names the quota and its usage. One is recorded for the operation’s result and one for its credential. Where the operation’s Job is gone, the window started from a foreground deletion of the record instead, and this result needs nothing more. The quota still has no room for the next operation’s records, though: it refuses that operation’s credential, with ResultCredentialFailed, and its result until these records age out. Where the Job still exists, the manager retries with backoff up to every 5 minutes until the quota has room or the Job is gone. Leave room for one retired record per operation, as A quota on result records says. |
RetiredApplyLogKept |
Warning | PtahSchema |
An execution binding change retired an Apply while it ran, and nobody will read its result. Its Pods have stopped, and the message says where anything the run wrote stays and for how long. |
RunnerProtocolMismatch |
Warning | PtahSchema, PtahMigration |
The runner from execution.runnerImage speaks another protocol than this manager and refused the Job before starting the executor. A read-only operation is tried again after the failure interval; an Apply still owes its read-only proof. |
SourceSnapshotRetirementFailed |
Warning | PtahSchema, PtahMigration |
A snapshot of an earlier ConfigMap source that nothing names any more could not be deleted. Nothing reads it, and the next resolution of the source, whatever its kind, deletes it. The message carries the API server’s refusal. |
TargetLockReleaseOwed |
Warning | PtahMigration |
The database lock was not released, and the release the claim owes will be retried. |
TerminationSummaryRefused |
Warning | PtahMigration |
A legacy Apply’s log held no readable result, and the summary in its Pod’s termination message was not read in its place. The message says why: another attempt, a summary cut short, or a log that holds a different frame. The run is recorded as unknown. |
UnresolvedRunAcknowledged |
Normal | PtahMigration |
A person’s PtahMigrationRunAcknowledgment settled the run this resource recorded as unresolved. The message names who, the outcome and the operation, and the resource reads its database again before it plans. |
UnresolvedRunDiscarded |
Warning | PtahMigration |
Deleting this resource discarded the record of a run nobody accounted for. The message names the outcome, the version, and the database. |
UnresolvedRunRestored |
Warning | PtahMigration |
The resource came back without the record of a run nobody accounted for in its status – a restore that dropped status – and the manager put it back from the copy in its operator.ptah.run/unresolved-run annotation. Nothing is planned until a reading or an acknowledgment settles it. |
VerificationPolicyInvalidated |
Warning | PtahSchema |
The verification policy stopped matching the plan, so artifact and plan verification were invalidated. |
An Event can be lost
Section titled “An Event can be lost”The API server keeps an Event for about an hour, and the manager’s recorder drops
some before they get there. It lets 25 Warnings through for each resource and then
one every five minutes, so a read that keeps failing through a long outage leaves
little for the Warning that follows it. A manager that stops just after a status
write sends nothing for that write. Read status when a decision depends on it:
status.journal is written in the status write that records the transition.
A Warning is not always a fault
Section titled “A Warning is not always a fault”Most of these say the operator refused to proceed, which is the behavior they
are there for: ProtectedTableRefused, ApprovalRevoked and PlanStale are
each a decision not to change a database on evidence that no longer holds.
The ones that ask for a person are MigrationRunUncertain,
ApplyOutcomeUnknown and TargetLockReleaseOwed: each says something about the
database is not yet established, and each has a page saying what settles it –
the mutation lifecycle for the first two,
and Operations for the lock.