v1.0.24 (P4b): canary drift alarm + incident snapshots #55

Merged

ClaudiuSchuster merged 1 commit from oss-oo/pinky-p4b-alarm-snapshots into main 2026-10-02 10:47:18 +00:00

Conversation 0 Commits 1 Files changed 12 +1247 -34

ClaudiuSchuster commented 2026-10-02 10:36:14 +00:00

Owner

• Copy link
• Copy Markdown

What (v1.0.24, roadmap P4b — MasterBrain steer A2-3)

Two observability mechanisms from the 2026-10-01 23:52 delivery-race
forensics — the capped ledgers (200/136/200) rotated out the
racer-side records before anyone could freeze them.

1. Incident snapshots — frozen ledger copies that survive rotation

• stream.py snapshot [--reason <text>] [--window <s>]: atomically
  copies the five ledgers (stream-dispatches/bells/pending/history +
  usage-ledger) into <config-dir>/incidents/<utc-ms>/ plus a
  manifest.json (files copied, record counts, reason, epoch,
  dispatcher version). Without --window the copies are byte-faithful
  raw freezes; --window <s> subsets by record timestamp (the family
  mixes epoch-seconds and epoch-ms fields — normalized on read).
• AUTO-trigger: the dispatcher's server-side-cancel detection (the
  v1.0.22 path) takes a snapshot automatically — fail-safe (never
  disturbs the exit path; suite-proven with a broken incidents path)
  and rate-limited to one per ZDO_SNAPSHOT_MIN_INTERVAL_S
  (default 300 s, cross-process: newest-dir check + flock — a cancel
  storm produces ONE dir per interval, suite-proven by counting dirs).
• Retention: every new snapshot prunes to the newest
  ZDO_SNAPSHOT_KEEP (default 20) incident dirs.
• New module snapshots.py (core + the DISPATCH_VERSION constant
  the manifest carries); wired into install.sh FILES + py_compile and
  the Makefile lists (the v1.0.17 new-module guard demands it).

2. Canary drift alarm — the alarm clock becomes an alarm

• stream.py canary-report [--rebaseline]: runs the contract canary's
  probe groups in their read-only mode and compares group outcomes
  against <config-dir>/canary-baseline.json:
  • exit 0 = no drift (quiet unless ZDO_CANARY_VERBOSE; a missing
    baseline is pinned on first run);
  • exit 1 = drift — zcode/infra/canary-drift published (plus the
    zcode/fleet/<station-id>/ mirror via publish_dual, NOT
    retained — it is an event) with payload
    {schema:"zcode/canary/v1", changedGroups, appVersion (the ZCODE_APP_VERSION derivation), timestamp}; changed groups printed;
  • exit 2 = usage/probe-infrastructure error (never silently green).
• contract_tests.py --probe (new flag): the read-only group subset —
  env/infra/db/bundle/fact only; no scratch sessions, no DB writes, no
  systemd-run units, no model requests; desktop notifications
  suppressed in this mode (the drift alarm is the alerting channel).
• Timer unit zcode-dispatch-canary.timer + .service
  (install.sh-templated like the bridge unit, explicit
  Environment="PATH=%h/.local/bin:…" per the station user-manager
  PATH lesson, hourly OnCalendar + Persistent=true) shipped
  DISABLED BY DEFAULT — the flip is the owner's/MasterBrain's move per
  the bridge precedent; runbook documents it. install.sh never runs
  systemctl (suite tripwire proves zero invocations).
• Canary discipline: canary-report relies on the SAME probe
  surfaces contract_tests already pins — no NEW upstream assumptions
  (the --probe mode only SELECTS the existing read-only groups; it
  adds no probes).

Suite + docs

• Suite legs (podman shim FIRST — zero real invocations, rig brokers,
  fake app-server cancel_race mode; never the live broker/units):
  snapshot content + manifest correctness; --window filtering
  (seconds/ms mix); retention pruning; auto-snapshot fires exactly
  once per interval under a rig cancel storm (dirs counted); broken
  snapshot path never disturbs the cancel exit; canary baseline pin →
  quiet exit 0; injected drift (probe-group outcome flipped via the
  rig's ZDO_CANARY_CMD seam) → exit 1 + the notice on BOTH topics
  asserted non-retained with fresh-connection no-replay;
  --rebaseline re-pins; new/vanished groups read as drift; probe
  exit 7 / garbage report → exit 2; install-leg canary unit-pair
  assertions + --live probe list.
• Full suite green (no regressions); super-linter green (station lane
  set).
• Docs: USAGE.md §16 (incident-snapshot runbook — what the dirs tell
  you, how they feed forensics, retention knobs) + §17 (canary-report:
  baseline lifecycle, exit codes, timer flip runbook);
  MQTT-TOPICS.md canary-drift row; README tools row + roadmap entry.
  PROMPT-APPENDIX.md untouched (agent-facing surface unchanged).

## What (v1.0.24, roadmap P4b — MasterBrain steer A2-3) Two observability mechanisms from the 2026-10-01 23:52 delivery-race forensics — the capped ledgers (200/136/200) rotated out the racer-side records before anyone could freeze them. ### 1. Incident snapshots — frozen ledger copies that survive rotation - `stream.py snapshot [--reason <text>] [--window <s>]`: atomically copies the five ledgers (stream-dispatches/bells/pending/history + usage-ledger) into `<config-dir>/incidents/<utc-ms>/` plus a manifest.json (files copied, record counts, reason, epoch, dispatcher version). Without `--window` the copies are byte-faithful raw freezes; `--window <s>` subsets by record timestamp (the family mixes epoch-seconds and epoch-ms fields — normalized on read). - AUTO-trigger: the dispatcher's server-side-cancel detection (the v1.0.22 path) takes a snapshot automatically — fail-safe (never disturbs the exit path; suite-proven with a broken incidents path) and rate-limited to one per `ZDO_SNAPSHOT_MIN_INTERVAL_S` (default 300 s, cross-process: newest-dir check + flock — a cancel storm produces ONE dir per interval, suite-proven by counting dirs). - Retention: every new snapshot prunes to the newest `ZDO_SNAPSHOT_KEEP` (default 20) incident dirs. - New module `snapshots.py` (core + the `DISPATCH_VERSION` constant the manifest carries); wired into install.sh FILES + py_compile and the Makefile lists (the v1.0.17 new-module guard demands it). ### 2. Canary drift alarm — the alarm clock becomes an alarm - `stream.py canary-report [--rebaseline]`: runs the contract canary's probe groups in their read-only mode and compares group outcomes against `<config-dir>/canary-baseline.json`: - exit 0 = no drift (quiet unless `ZDO_CANARY_VERBOSE`; a missing baseline is pinned on first run); - exit 1 = drift — `zcode/infra/canary-drift` published (plus the `zcode/fleet/<station-id>/` mirror via `publish_dual`, NOT retained — it is an event) with payload `{schema:"zcode/canary/v1", changedGroups, appVersion (the ZCODE_APP_VERSION derivation), timestamp}`; changed groups printed; - exit 2 = usage/probe-infrastructure error (never silently green). - `contract_tests.py --probe` (new flag): the read-only group subset — env/infra/db/bundle/fact only; no scratch sessions, no DB writes, no systemd-run units, no model requests; desktop notifications suppressed in this mode (the drift alarm is the alerting channel). - Timer unit `zcode-dispatch-canary.timer` + `.service` (install.sh-templated like the bridge unit, explicit `Environment="PATH=%h/.local/bin:…"` per the station user-manager PATH lesson, hourly `OnCalendar` + `Persistent=true`) shipped DISABLED BY DEFAULT — the flip is the owner's/MasterBrain's move per the bridge precedent; runbook documents it. install.sh never runs systemctl (suite tripwire proves zero invocations). - **Canary discipline**: canary-report relies on the SAME probe surfaces contract_tests already pins — no NEW upstream assumptions (the `--probe` mode only SELECTS the existing read-only groups; it adds no probes). ### Suite + docs - Suite legs (podman shim FIRST — zero real invocations, rig brokers, fake app-server cancel_race mode; never the live broker/units): snapshot content + manifest correctness; `--window` filtering (seconds/ms mix); retention pruning; auto-snapshot fires exactly once per interval under a rig cancel storm (dirs counted); broken snapshot path never disturbs the cancel exit; canary baseline pin → quiet exit 0; injected drift (probe-group outcome flipped via the rig's `ZDO_CANARY_CMD` seam) → exit 1 + the notice on BOTH topics asserted non-retained with fresh-connection no-replay; `--rebaseline` re-pins; new/vanished groups read as drift; probe exit 7 / garbage report → exit 2; install-leg canary unit-pair assertions + `--live` probe list. - Full suite green (no regressions); super-linter green (station lane set). - Docs: USAGE.md §16 (incident-snapshot runbook — what the dirs tell you, how they feed forensics, retention knobs) + §17 (canary-report: baseline lifecycle, exit codes, timer flip runbook); MQTT-TOPICS.md canary-drift row; README tools row + roadmap entry. PROMPT-APPENDIX.md untouched (agent-facing surface unchanged).

ClaudiuSchuster added 1 commit 2026-10-02 10:36:14 +00:00

v1.0.24 (P4b): canary drift alarm + incident snapshots ... dc8da91f95

Two observability mechanisms from the 2026-10-01 23:52 delivery-race
forensics (the capped ledgers rotated the racer-side records away
before anyone could freeze them):

- incident snapshots: stream.py snapshot freezes the five ledgers
  under <config-dir>/incidents/<utc-ms>/ (byte-faithful raw copies, or
  --window-filtered record subsets across the family's seconds/ms
  timestamp mix) with a manifest (files, counts, reason, epoch,
  version); the dispatcher's server-side-cancel detection (v1.0.22 D2)
  auto-freezes one dir per ZDO_SNAPSHOT_MIN_INTERVAL_S (default 300,
  cross-process via newest-dir check + flock — a cancel storm must not
  fill the disk), fail-safe wrapped so the fast-fail exit is never
  disturbed; retention keeps the newest ZDO_SNAPSHOT_KEEP (default 20)
  dirs. New snapshots.py module (core + DISPATCH_VERSION constant).

- canary drift alarm: stream.py canary-report runs the contract
  canary's read-only probe groups (new contract_tests.py --probe mode:
  env/infra/db/bundle/fact only — no scratch sessions, no DB writes,
  no systemd-run, no model requests; the SAME probe surfaces the
  canary pins, no new upstream assumptions), rolls each group's checks
  to one outcome (fail > warn > skip > pass) and compares against
  canary-baseline.json: exit 0 quiet (missing baseline pinned on first
  run), exit 1 drift (changed groups printed + zcode/infra/canary-drift
  dual-published non-retained, schema zcode/canary/v1 with
  changedGroups/appVersion/timestamp), exit 2 probe-infrastructure
  error — never a silent green. --rebaseline re-pins. Hourly systemd
  timer + service (explicit Environment= PATH) shipped DISABLED by
  default (the bridge precedent; runbook in USAGE.md).

Suite legs: snapshot content/manifest/window/missing/retention;
auto-snapshot fires exactly once per interval under a rig cancel
storm + broken-path fail-safe; canary baseline/drift/rebaseline/
new+vanished groups/exit-2 with rig-broker MQTT assertions on both
topics (non-retained, fresh-connection no-replay); install-leg canary
unit pair assertions + --live probe list. Docs: USAGE.md sections
16-17, MQTT-TOPICS.md canary-drift row, README row + roadmap entry.

ClaudiuSchuster merged commit 0f11778a15 into main 2026-10-02 10:47:18 +00:00

ClaudiuSchuster referenced this pull request from a commit 2026-10-02 10:47:19 +00:00

v1.0.24 (P4b): canary drift alarm + incident snapshots (#55)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

OSS-Singularity/zcode-dispatch!55

No description provided.