zcode-dispatch

Dispatch and steer ZCode sessions (GLM models) from any external agent — Claude Code, Codex, custom orchestrators — using the official zcode-cli app-server protocol. No GUI interaction, no headless-mode model limits, no hacks.

But that one-line pitch undersells what grew here. zcode-dispatch is the muscle of a multi-agent orchestration: children dispatched with --notify-parent ring a 🔔 doorbell digest back into the parent session when they finish (busy parents queue — the sweeper delivers when idle, never dropped, with separate rate budgets for bells and owner steers and a watchdog that escalates stuck bells); --notify-cc rings bell-only CC copies of the same digest into further sessions — observers and deep trees fan out without stealing the parent's result ownership (v1.0.19); every turn publishes events onto an MQTT bus (zcode/session/<id>/turn-completed · -failed · -timeout, plus bell and dispatch lifecycle topics), with retained per-session state snapshots (zcode/session/<id>/state — restarted consumers get instant state without a session-DB rescan, archived tombstones prune) and fleet mirrors (zcode/fleet/<station-id>/…, dual-published during the transition — v1.0.18); pending AskUserQuestions park for external answering; a bell ledger records every spawn so whole mission trees stay visible; and a dispatch lifecycle ledger plus a silent-death reaper flag workers whose process died without a word (the firing parent gets the trail automatically). Pair it with the zcode-dispatch-observatory cockpit and you can watch, steer, and answer all of it from one page — see Pair it with the observatory.

The orchestration — the pinky brotherhood

Central Brains dispatch detached worker Pinkies; the workers ring doorbells back on return; everything publishes turn events onto an MQTT bus; safety nets (sweeper, watchdog) catch what a bell misses; and the observatory cockpit reads it all and reaches back in (steer, answer). Validations close the loop through PRs into releases and installs back onto the station:

flowchart TB
    subgraph BRAINS["🧠 Brains — central ZCode chats"]
        BRAIN["mission brain<br/>open desktop chat"]
    end
    subgraph PINKIES["🤖 Pinkies — dispatched workers (detached)"]
        direction LR
        P1["worker pinky<br/>repo lane"]
        P2["worker pinky<br/>gate lane"]
    end
    subgraph BELLS["🔔 Bells + MQTT — the nervous system"]
        direction LR
        SWEEPER["sweeper queue<br/>retry + rate budgets"]
        WATCHDOG["watchdog<br/>stuck-bell alerter"]
        MQTT["MQTT broker — podman mosquitto<br/>restart=always + watchdog timer"]
        LISTENER["stream.py listen<br/>zcode/session/+/+"]
        EXT["external consumers<br/>scripts · automations"]
    end
    subgraph COCKPIT["🔭 Observatory — the cockpit"]
        direction LR
        DB[("ZCode session DB<br/>(read-only)")]
        OBS["dashboard · lineage colors<br/>transcript · response modal"]
        STEER["steer console · question cards<br/>archive · pin · favorites dock"]
    end
    subgraph SHIP["🚢 Release-return"]
        direction LR
        PR["PR → checks green<br/>→ squash-merge"]
        REL["tag + release"]
        INSTALL["station install<br/>~/tools"]
    end

    BRAIN -->|"dispatch (detached zcode-dispatch)"| P1
    BRAIN -->|"dispatch"| P2
    P1 -->|"🔔 doorbell digest on return"| BRAIN
    P2 -->|"🔔 doorbell (parent busy → queues)"| SWEEPER
    SWEEPER -->|"delivers verbatim when idle"| BRAIN
    WATCHDOG -.->|"🔔 stuck alert (> 5 min)"| BRAIN
    P1 & P2 -->|"turn-completed / -failed / -timeout"| MQTT
    MQTT --> LISTENER
    MQTT -.->|"pub/sub, any consumer"| EXT
    BRAIN & P1 & P2 --> DB
    DB -->|"live rows · lineage · bell states"| OBS
    STEER -->|"steer / answer (spawns zcode-dispatch)"| BRAIN
    P2 -->|"push branch"| PR
    PR --> REL
    REL --> INSTALL
    INSTALL -.->|"next dispatch rides the fresh tools"| BRAIN

Every arrow is a real, verifiable mechanism of this repo: the dispatch preflight and exit-code contract below, the doorbell digest and queue semantics in docs/USAGE.md §8, the full topic layout (including notification-delivered, bell-stuck, presumed-dead, reaper-escalation) in docs/MQTT-TOPICS.md.

How one dispatch runs

flowchart TB
    AG["🧠 external agent<br/>Claude Code · Codex · any orchestrator"]

    subgraph PRE["⚙ preflight — one driver per cwd"]
        direction LR
        LOCK["cwd lock · PID-checked<br/>collision → exit 3"]
        CFG["config · pinned model + reasoning<br/>plan key decrypted in-process (AES-GCM)"]
        GATE{"new or<br/>steer?"}
        LOCK --> CFG --> GATE
    end

    subgraph BRIDGE["🔌 app-server bridge — stdio · NDJSON"]
        direction LR
        MIRROR["provider/updateAccountConfig<br/>entitlement mirror"]
        CREATE["session/create<br/>static title pinned"]
        RESUME["session/resume<br/>cwd · model · reasoning verified"]
        SEND["session/send {content}"]
        IACT["runtime prefs + provider headers<br/>answered for own provider/model only"]
        POLL["session/usage + turn_usage<br/>polled until terminal"]
        MIRROR --> CREATE --> SEND
        RESUME --> SEND
        SEND --> IACT --> POLL
    end

    DONE{"turn<br/>outcome"}
    JSON["📤 sanitized JSON → --out · incremental<br/>actual model + reasoning verified from the store"]

    subgraph DESK["🖥 desktop — persistent · click-through"]
        direction LR
        NSTART["🚀 STARTED notify<br/>New · Steer"]
        NEND["✅ COMPLETED · ❌ FAILED · 🟡 TIMEOUT<br/>click → zcode:// workspace deep link"]
        VIS["task-index seeding — show_sessions.py<br/>sidebar 'By project' visibility"]
    end

    AG -- "prompt file" --> LOCK
    GATE -- "new" --> MIRROR
    GATE -- "steer · --resume" --> RESUME
    SEND -. "turn starts" .-> NSTART
    CREATE -. "title = sidebar = notify" .-> VIS
    POLL --> DONE
    DONE -- "exit 0" --> JSON
    DONE -- "exit 4" --> JSON
    DONE -- "exit 5 · resumable" --> JSON
    DONE --> NEND
    JSON -. "polled read-only" .-> AG

    classDef neutral fill:#10141c,stroke:#8a97a4,stroke-width:1.5px,color:#e8f0f6
    classDef server fill:#10141c,stroke:#4f9cf5,stroke-width:1.5px,color:#e8f0f6
    classDef session fill:#0e2a2f,stroke:#4fe3c1,stroke-width:1.5px,color:#e8f0f6
    classDef desk fill:#191326,stroke:#f0b95a,stroke-width:1.5px,color:#f4e3c2
    classDef result fill:#123a52,stroke:#62c7f5,stroke-width:2px,color:#ffffff
    class AG,LOCK,CFG neutral
    class MIRROR,IACT,POLL server
    class CREATE,RESUME,SEND session
    class NSTART,NEND,VIS desk
    class JSON result
    style PRE fill:#0b0e14,stroke:#22303e,color:#c3cdd6
    style BRIDGE fill:#0d2836,stroke:#4fe3c1,stroke-width:2px,color:#eafff8
    style DESK fill:#120f1a,stroke:#f0b95a,color:#f4e3c2

Why

"Wir bauen nicht mehr ein Produkt — wir bauen ein NERVENSYSTEM." "We are no longer building a product — we are building a nervous system."

It started as two upstream gaps — zcode-cli -p (headless) had no --model flag, and app-server sessions were invisible in the desktop app — and the dispatcher closed both (explicit model + reasoning selection, account-state mirroring, runtime-auth interaction handling, visible-session seeding; the forensics live in docs/PROTOCOL.md).

It became the brain-pinky orchestration platform: the doorbell protocol, delegation (--child-role brain — sub-brains running their own pinkies), the validation flywheel (every round ships through PRs, checks and live drills), the contract canary that alarms when upstream changes break an unpaid assumption, the MQTT event bus every turn publishes onto, and the lifecycle reaper that names silent deaths.

It enables the combination with the observatory — steer, answer, monitor and orchestrate every session from one cockpit. The vision it serves: a multi-brain orchestra with latency to zero, pinkies as the owner's extended arms, brains as extensions of the owner's vision. Not marketing speak — lived truth, one validated round at a time.

The original two-gap solver is still all there, one paragraph up: explicit model + reasoning selection (session/create), account-state mirroring (provider/updateAccountConfig) so the plan providers materialize, runtime-auth interaction handling (interaction/requestProviderRuntimeHeaders) with the plan credential decrypted in-process only, visible sessions (task-index seeding + static pinned session titles), and sanitized JSON results with verified (not echoed) model/reasoning/usage.

Installation

oo-dispatch — the OO lane's dispatcher: a companion alias of (and the former name-in-progress rename of) zcode-dispatch, installed as a second symlink to the same dispatch.py (nothing renamed, v1.0.88).

From a release tarball (the exact released file set; see the releases page for the latest version):

curl -L -o zcode-dispatch.tar.gz \
  https://oss-oo.io/OSS-Singularity/zcode-dispatch/releases/download/vX.Y.Z/zcode-dispatch.tar.gz
tar xzf zcode-dispatch.tar.gz
cd zcode-dispatch
./install.sh --target ~/tools/zcode-dispatch

install.sh copies the scripts and docs, preserves an existing config.json (so re-running it after a release is an update-in-place), and symlinks the dispatcher as zcode-dispatch (plus the companion commands) into ~/.local/bin (--target, --bin, --no-bin to customize). It also installs the zcode-events-bridge systemd user unit — disabled by default; enabling it is the coordinated station cutover step (docs/EVENTS.md, --no-systemd-units to skip). From a git clone you can also run the scripts in place (./dispatch.py …) — nothing else is required beyond Python 3.10+ and pip install cryptography.

Verifying an install (install.sh --check)

The station acceptance triple — installed bytes match the source, symlinks resolve into the install, exec bits present — is a first-class command (v1.0.21). Run it from a source checkout against an installed prefix:

./install.sh --check --target ~/tools/zcode-dispatch --against vX.Y.Z   # vs a tag
./install.sh --check --target ~/tools/zcode-dispatch                    # vs origin/main (fetched best-effort)

It prints one line per check group: (1) every payload file on the installer's FILES list is byte-identical to the comparison source, (2) the FILES list still covers dispatch.py's local imports (the packaging guard), (3) every command symlink the installer owns resolves into the install and dangles nowhere, (4) the runtime entry points carry their exec bits (the status=203/EXEC class). Comparison sources: origin/main by default, --against <dir> for a plain directory (offline rigs), --against <tag-or-ref> for any git ref of the checkout — an unresolvable source is a clear error, never a silent skip. Exit codes: 0 green, 1 drift (the drift list names the file/symlink/bit), 2 usage or comparison-source error. --live adds read-only systemctl --user is-enabled/is-active reports for the sweep and watchdog timers and the events bridge — a station observation, not install correctness; it never mutates anything. Full reference: docs/USAGE.md §15.

Quick start

pip install cryptography        # AES-GCM for the credential store
./dispatch.py --setup          # auto-detect provider/models/credentials → config.json
./dispatch.py \
  --cwd /path/to/exclusive/worktree \
  --model GLM-5.3 \
  --prompt-file task.md \
  --out result.json --timeout 7200

(--setup is optional — copying config.example.json to config.json and editing it works just as well.)

Prerequisite: zcode-cli login (Z.AI OAuth) done once — the plan credential is read from the shared ZCode credential store and stays in dispatcher memory only. Full reference (flags, exit codes, result schema, protocol internals, detached launches, recovery): docs/USAGE.md.

Flags

Flag	Meaning
--cwd	exclusive working directory (worktree recommended); a path under /tmp is auto-rewritten to the durable mirror ~/.zcode/tmp/<path-after-/tmp> (created as needed, loud: stderr note + effective cwd + cwdNotes in the result), and a resume whose remembered cwd is a dead /tmp path auto-heals — mirror created, the line's files moved, the session row updated, the turn continues there; ZDO_ALLOW_TMP_CWD=1 keeps /tmp exactly as given (never silent); the teaching refusal fires only when healing is impossible (v1.0.75, docs/USAGE.md §32)
--model	one of the configured models (default: GLM-5.3, GLM-5.3-Flash)
--prompt-file	UTF-8 prompt; sent as one turn (content field)
--out	result JSON path (written incrementally — poll for read-only monitoring)
--resume sess_…	continue an existing session (cwd/model/reasoning verified against the store); an uncredentialed OAuth-individual pin falls back LOUDLY to the config provider (v1.0.75 PINFALLBACK — turn runs on team, stderr NOTE + pinFallback result field, both runways named; keyless-everything stays an honest exit-6 naming both)
--notify-parent sess_…	wake that parent session with a 🔔 digest turn when this child finishes (busy parents queue for stream.py sweep — never dropped, TTL + dead-letter hygiene)
--no-parent-bell	OPT OUT of the per-session bell contract (v1.0.56): without it, a --resume/steer dispatch of a child whose create carried --notify-parent inherits the persisted parent edge and rings it at terminal state — notification deliveries never inherit (no grandparent cascade); an explicit --notify-parent still wins
--no-bell, --fire-and-forget	DECLARED-INTENT opt-out of the bell doctrine (v1.0.75, UNGELOCKT): an agent-env child create without --notify-parent raises the unbelled-spawn alarm by default (loud log line, deck event, desktop notification, receipt marker) — this flag is the ONLY silence, recorded who/when on receipt + lifecycle ledger; implies --no-parent-bell, refused with --notify-parent (exit 2); see docs/USAGE.md §30
--notify-cc sid[,sid…]	ring additional sessions with bell-only CC copies of the same doorbell digest (observers; v1.0.19): entries stripped/deduped, parent-equal entries dropped, max 16 after dedupe, unknown ids refused with exit 2; legal without --notify-parent (sender = this session)
--cc-on {all,failure}	when --notify-cc bells fire (v1.0.19): all (default) = completed/failed/timeout; failure = failed/timeout only — the parent bell always fires on all three
--wake-in MIN	delayed self-wake fallback (v1.0.77 DIREKTFLUSS): at EVERY terminal state of this turn (completed/failed/timeout alike — an alarm clock rings regardless of how the turn ended) park a one-shot wake timer of MIN minutes on THIS dispatch's session; the sweeper's wake pass rings ⏰ wake-in timer elapsed through the normal bell machinery (busy self-queue, receipts, exactly-once; the 15 s sweep tick is the precision). The event-source watchers (ci-watch/unit-watch) are the primary layers; this is the floor beneath them. Bounded by wake.wake_in_max_min (default 720, hard bound 30 days); stream.py wake-in --session … --in … --note … is the after-the-fact twin for any lane
--watch-pr OWNER/REPO#N	spawn-side CI watch (v1.0.77 DIREKTFLUSS): at this turn's terminal state register the forge CI watch for the named pull request on THIS dispatch's session — the sweeper's ci_watch pass (config-gated wake.ci_watch.enabled) rings ONE bell when the PR's CI ends (verdict in the prompt) or the PR merges/closes; the lane that shipped the PR registers its own wake at birth. stream.py ci-watch --repo … --pr … --session … is the after-the-fact twin; malformed specs refuse at parse time (exit 2)
--child-role {pinky,brain}	role of this child in the delegation topology (default pinky = worker); brain marks a delegation — a sub-brain expected to run its own orchestra of pinkies — and stamps childRole into the result JSON, a [🧠] prefix onto the pinned static title, role into the bell + lifecycle ledgers, and 🧠 brain-child wording into the doorbell prompt
--timeout	seconds until exit 5 — no timeout by default (default_timeout: 0 in the config); opt into a limit for bounded runs (min 20 s)
--systemd	re-exec the whole dispatch into its own systemd user scope and return immediately, fire-and-forget (v1.0.34): one {unit, resultFile, note} JSON line + exit 0 while the scoped dispatcher writes the normal incremental result JSON (scope names the unit) — survives the spawner's exec-cgroup teardown at turn end (the cgroup-murder class; replaces the manual systemd-run dance); unit zdispatch-<timestamp>-<cwd-slug>-<nonce>, env-complete re-exec (PATH/HOME + ZCODE_*/ZDO_* seams), --same-dir for relative argv, honest exit 2 when the scope cannot start, logged no-op in test mode / ZDO_NO_SCOPE_REEXEC=1 (the opt-out wins)
--mode	yolo (default) / build / edit / plan
--turn-input {soft,interrupt}	the AKTIV-TURN-INPUT contract at a BUSY mid-turn target (v1.0.75 HAMMER): soft (default; flag > ZDO_TURN_INPUT > config turnInput) NEVER touches the running turn — queue-until-idle stands; interrupt (opt-in) performs the visible stop-then-send (session/stop first, effect verified by transcript quiet-probe, [turn-input: …] annotation in-band + turnInterrupt in the result JSON; an ineffective stop — the cross-process no-op, bundle-proven — re-queues honestly as queued-busy; one attempt, one driver)
--turn-nudge	deliver mid-turn against a busy target and let the app-server HOLD the input to the next turn boundary (the documented protocol hold; soft by construction — never a cancel; sweep eligibility ZDO_TURN_NUDGE_S, default off); visibility companions: stream.py queue-peek and the rung-gated steer-deferred MQTT event
OAuth plan ride (no flag)	v1.0.75 HAMMER: a provider with NO stored api-key but an OAuth ride (auth-source: oauth + identity in the shared credential store) answers interaction/requestProviderRuntimeHeaders with a SHORT-LIVED PROJECT TOKEN minted through the desktop client's own chain (business login → default org/project → zcode-api-key → access_tokens mint; oauthmint.py) — headless OAuth plans without stored keys; manual keys always win; mint failures decline honestly (never a wedge); ZDO_OAUTH_MINT=0 restores the pre-HAMMER exit 6; rig seam ZDO_OAUTH_ORIGIN
--reasoning	low / high / max (default from config, usually max)
--budget-override	bypass a HARD budget refusal for this one dispatch (anti-starvation — the owner stays in command); the override self-audits on every use (v1.0.76 OVAAUDIT): one kind: "override" event in the usage ledger + one override field in the result JSON, even when no guardrail fired and with usage accounting off (v1.0.16); since v1.0.75 (STEERFELDER) the marker rides steers and an overridden child's MANDATORY digestion bell end-to-end — self-queue, watchdog rescue, reclaim and the sweep's re-drive all re-fire the SAME audited hatch, so the bell is never budget-refused (docs/BUDGETS.md "Override parity")
--override-auth REF	owner-authorization reference recorded with the --budget-override self-audit (v1.0.76): path to or identifier of the OWNER-AUTHORIZATION.json that cleared the override (env ZDO_OVERRIDE_AUTH as fallback); the dispatcher records the reference, never reads the file
--setup	detect provider/models/credentials from the installed ZCode runtime, write the config, exit
--plan-switch SESS_ID PROVIDER	standalone mode (v1.0.75, STEERFELDER §6): re-pin an EXISTING session onto another plan — surgically rewrites BOTH pin surfaces the resume path reads (session_entry runtime/model_selection + the newest message stamp) and appends ONE kind=plan-switch audit event; --plan-model, --config-default, --allow-uncredentialed tune it; refusals (exit 2): unknown sid, RUNNING dispatcher, OAuth-class target without --allow-uncredentialed (docs/USAGE.md §31)
--list-providers	standalone (v1.0.75): print the provider-capability truth (keyed vs OAuth class, the config default) the observatory's "Plan wechseln" renders (docs/USAGE.md §31)
--usage-summary	print the rolling-window usage summary (per-lane + total token sums, JSON) from the usage ledger, refresh the retained MQTT usage rollups, exit — the read side of the budget accounting
--kiss <MILESTONE>	the Mama kiss (v1.0.82): on a COMPLETED round, fire ONE milestone line [zcode-dispatch 📡] <MILESTONE> into Mama's Codex thread via the sanctioned read-only seam codex exec resume <thread> --skip-git-repo-check -s read-only — throttled by kiss.min_interval_s (default 900 s; a kiss inside the window DROPS with a note, never queues), armed by kiss.enabled + kiss.mama_thread_id (OFF by default; ZDO_KISS_ENABLED/ZDO_KISS_THREAD/ZDO_KISS_MIN_INTERVAL_S env overrides), receipts in kiss-receipts.jsonl (config dir), result carries kissAttempted/kissDropped/kissExit, fail-safe (a kiss problem never fails the round); failed/timeout rounds never kiss (docs/USAGE.md §34). v1.0.83 (CADAPTER): parent-CLI-agnostic — kiss.targets[] names one or more targets with a backend each: codex (the seam above, argv byte-identical) or claude (Claude Code headless into a REGISTERED postbox session: <bin> -p --resume <session> --output-format json …, a BOUNDED argv — no tools, one turn, no permission prompts, no MCP, a dollar cap); receipt truth per backend (claude: exit 0 AND stdout JSON AND is_error false AND session_id == target), targets fire independently with their own throttle, the result adds kissDelivered + kissTargets; a malformed kiss config refuses with exit 2 BEFORE any spawn (docs/USAGE.md §34.1) Since v1.0.87 the codex seam waits up to 300 s for the turn (per-target timeout_s overrides) and every receipt carries timeout_s. v1.0.94 (APPSERVER-TRANSPORT): backend appserver — the EXTERNAL transport oo-appserver (≥ v0.2.0, another repo; one contract, two repos): the dispatcher CALLS <bin> send --socket <path> --thread <id> (bin + socket + thread in the target; the line rides the child's STDIN — secrets never argv) and MAPS its JSON receipt: delivered-headless/delivered-app-start → delivered, delivered-app-steer → steered (with the documented turn-abort caution), held-by-app → the NAMED refusal (caution note documented — the app holds the thread, not quota/drift/dead), heal-skipped → note-only (the transport's reason quoted verbatim); the five classes live ALONE in oo-appserver, default clock 60 s (kiss_appserver.py, docs/USAGE.md §34.2). The equal-rights family path (owner doctrine 2026-10-28): oo-appserver send <thread> <line> is a FIRST-CLASS delivery path, not a fallback — families may call it directly. The LAUNCH LAW: doors into the owner's app-server ONLY via this channel — never CDP, never UI injection, only with the owner's explicit yes per experiment. v1.0.95 (CLAUDE-SESSION): the fourth backend claude-session — the uds door into a RUNNING Claude Code session (claude_door.py, registry ~/.claude/sessions, matched by sessionId/pid/name): no subprocess, no model turn; NAMED receipt classes accepted/no-session/refused/socket-error, the registry status rides as a hint (busy=steer-like, idle=start-like), env ZDO_CLAUDE_SESSIONS_DIR for hermetic rigs (docs/USAGE.md §34.3). The accepted truth: delivery to the model is UNCONFIRMED — the connection was held open, the receiver may hold or expire the peer line. v1.0.110 (CLAUDE-INBOX, CONTRACT v1): the fifth backend claude-inbox — THE PROVEN PATH: one line [zai 📡] <milestone> appended to ~/.zcode/tmp/claude-inbox/<sessionId>.inbox (target = session/name; a full sessionId works registry-free); the DOORMAN heartbeats <inbox>.doorman.json and acks <inbox>.ack with the line's sha256 — the sender is delivered (delivered-steer/delivered-start) ONLY with the sha-matching ack, else timeout (exit 5) / queued-no-doorman / queued-offline (exit 3), the line always waits in the file (nothing is lost). The .96–.109 carrier path is DOWNGRADED to accepted-only (NEVER delivered — the carrier process tree carries no foreign lines to the model, Claude's live measurement; the postbox turn cost $0.51 at the $0.05 cap and delivered nothing — issue #148 closes with this round) (docs/USAGE.md §34.4).
--kiss-test	standalone (v1.0.82): fire ONE configured test line through the REAL backend seam — the only real-call path (arming probe, owner-blessed); since v1.0.86 a dry argv probe against the installed codex CLI runs first (bogus thread id, no turn, no quota) and CLI drift refuses with exit 2; refused exit 2 unarmed or on a malformed kiss config (nothing sent), 0 when EVERY selected target verifiably delivered / 4 otherwise, throttle bypassed by design Since v1.0.89 every kiss child reads /dev/null and a codex receipt names thread-held (the ChatGPT app holds the thread) or timeout.
--kiss-target NAME	restrict --kiss / --kiss-test to the named kiss target(s) (v1.0.83 CADAPTER; repeatable; an unknown name refuses with exit 2 before any spawn)

Exit codes: 0 completed · 2 arguments · 3 concurrency/resume mismatch · 4 turn failed (a mid-turn SSE drop closed by the v1.0.75 SILENTDEATH watchdog carries error_class: stream-drop — the weather-cure class; see "The weather cure" below; v1.0.75 RESETSYNC: a cleanly-classified death parks its own continuation — quota deaths and 0-token pinned-signature transport deaths sleep in stream-revivals.json and auto-resume the SAME session, bell suppressed; error_class names the class: quota-exhausted | network-transport; see docs/USAGE.md §30) · 5 timeout (session intact, resumable) · 6 bridge/config error, or landing-strip (v1.0.74 LANDEBAHN: a quota window's remaining credits are at/below the landing strip — the ABSOLUTE budget for 1-2 clean wrap-up turns, the orchestration's only quota admission control — in NON-full-speed modes: the switch kills the strip (full-speed ON forces 0, pure burning — the owner's fifth reading); refused at admission; error_class landing-strip distinguishes; --budget-override passes audited; the strip lifts at the observed window's reset — controlled re-entry; v1.0.75 RESETSYNC: a refused queued delivery parks a first-class strip record the reconciler re-arms ON the reset event and REBUILDS from when the queue row vanished — an empty waiting queue proves nothing, exactly-once holds across the revival race, and the accepted vs rd-… re-delivery ids stay distinct and stable on every ledger surface; see docs/USAGE.md §22/§30) · 7 budget-refused (hard token limit exceeded at admission — no session, no spawn; see docs/BUDGETS.md). The v1.0.59/v1.0.66 exit-8 pace-deferred/lane-parked families are retired (v1.0.74: the pace projection is ADVISORY — a warning that never defers; the strip refusal is the park shape; §24/§28).

End-verification honesty (v1.0.75 REGRESSFOLD). A completed turn's model/reasoning is verified against what was actually dispatched — with one truth-faithful exception: a resume whose turn left no reasoningLevel stamp (pre-RESLEV sessions) is completed with a reasoningUnproven note in the result JSON and lean print, never a false failed that obscures the delivered answer — a missing stamp is unprovable, not mismatched (creates keep the strict axis; a resume that does stamp a disagreeing level still fails). Superseded verdicts state the matched delivery's shape: every terminal dispatch stamps its send-ack receipt (delivery_outcome + delivery_answer), so an identical-prompt retry after a verification-failed delivery still supersedes once (ONCE semantics byte-intact) but the verdict carries matchedDelivery and the note delivered stands, verification unproven — retry only if that answer is unusable.

Pair it with the observatory (the dreamteam)

zcode-dispatch-observatory is the cockpit this muscle deserves — and it rides on this repo's own architecture. The dispatcher is independently useful (any external agent, plain files, plain exit codes), and so is the observatory (it sees and interacts with every ZCode session, desktop chats included); together each covers the other's blind spots:

• Steering UI — the observatory's console spawns this dispatcher (--resume, detached) to run follow-up prompts, with the same queue-until-idle semantics, an optimistic echo of your words, and live queue depth. No terminal, no agent prompt needed to steer a session.
• Question answering — AskUserQuestions parked by this dispatcher's side-channel (ZCODE_DISPATCH_ANSWERABLE=1) are answered from the observatory's question cards: click an option, the turn continues. Tool-level permission requests ride the same family since v1.0.75 (kind: "tool-approval" cards: grant releases the turn, deny terminalizes honestly — the attempt2 blind class closed).
• Archive, pin, favorites dock — the observatory drives the desktop's own archive store and keeps chats at hand; dispatch work becomes triage-able like any other session.
• Lineage visibility — the bell ledger and doorbell edges this repo writes become whole mission trees, color-coded by root family, with pending/stuck bell states on the parent rows.
• Bell transparency — the observatory's busy-state hints mirror this dispatcher's own precheck signals (transcript activity grace), so "queued · waiting for idle" means exactly what the sweeper will do.

Dispatch from anywhere; watch, steer, and answer from one page.

Companion tools

Tool	Purpose
open_session.sh <sessionId>	Deep-link a session's workspace into the ZCode desktop app
show_sessions.py	Insert unindexed sessions into the desktop app's task index (visibility without deep links)
notify_click.sh	Persistent clickable desktop notification (used by the dispatcher); the click opens the OBSERVATORY at that session (?session=<id>, port-probed, ZDO_OBSERVATORY_URL overridable) with the ZCode deep link one click away
fake_app_server.py	Protocol fake for offline tests of error/timeout paths
quota.py	Z.AI quota monitor API home (v1.0.49): key resolution chain (env ZDO_QUOTA_API_KEY > config quota_api_key > the coding-plan key file ~/.api-tokens/zcode-dispatch-cp.z-ai.api.key — the arming class; the legacy standard-class path is a deprecated read-ok spare, never consulted), the --quota live-status verb (5 h + 7 d windows, human + --json) and --set-quota-key (validate-before-persist, atomic 0600 write, masked confirm) — the contract is docs/QUOTA.md
stream.py	Narf-doorbell sweeper (sweep/status/listen) + MQTT event bus client + contract-canary version watch + MQTT broker watchdog (mqtt-watchdog, v1.0.1) + incident snapshots (snapshot, v1.0.24) + canary drift alarm (canary-report, v1.0.24) + dead-dispatcher title reconciliation in every sweep + one-time legacy title heal (title-heal, v1.0.33) + voluntary quota parks (quota-park, v1.0.40) + surgical bell-contract seam (bell-parent: re-parent/clear a session's bell contract, retire/unretire dead bell targets, audit live sessions' parents — v1.0.65, docs/USAGE.md §27) + DIREKTFLUSS wake layers (v1.0.77): wake-in delayed self-wake timers, unit-watch systemd unit-maturity bells, ci-watch forge CI-end bells (registration-based, config-gated, TTL-bounded) + idle-fleet watchdog in every sweep (v1.0.75 IDLEWATCH: a fully-idle fleet inside an ACTIVE window screams at the master — bell + owner-care popup + deck event, backoff re-fires, honest dormancy; the EXPLICIT-PARK amendment adds the idlewatch-park seam for deliberate waiting — docs/USAGE.md §35) + background registry (v1.0.99 BG-REGISTRY: every initiated background work registers in <config-dir>/bg-registry.jsonl — open row per registration, close row per end, append-only; CLI bg-register/bg-close/bg-list [--active], automatic rows for unit-watch/ci-watch/--watch-pr and for every sweep tick — the crash traceability + the idle false-alarm read endpoint) + per-session turn-state view (turn-state: model-step class, active tool, approval wait, quota park, terminal — v1.0.75 STEERFELDER, docs/USAGE.md §30)
stream.py	Narf-doorbell sweeper (sweep/status/listen) + MQTT event bus client + contract-canary version watch + MQTT broker watchdog (mqtt-watchdog, v1.0.1) + incident snapshots (snapshot, v1.0.24) + canary drift alarm (canary-report, v1.0.24) + dead-dispatcher title reconciliation in every sweep + one-time legacy title heal (title-heal, v1.0.33) + voluntary quota parks (quota-park, v1.0.40) + surgical bell-contract seam (bell-parent: re-parent/clear a session's bell contract, retire/unretire dead bell targets, audit live sessions' parents — v1.0.65, docs/USAGE.md §27) + DIREKTFLUSS wake layers (v1.0.77): wake-in delayed self-wake timers, unit-watch systemd unit-maturity bells, ci-watch forge CI-end bells (registration-based, config-gated, TTL-bounded) + idle-fleet watchdog in every sweep (v1.0.75 IDLEWATCH: a fully-idle fleet inside an ACTIVE window screams at the master — bell + owner-care popup + deck event, backoff re-fires, honest dormancy; the EXPLICIT-PARK amendment adds the idlewatch-park seam for deliberate waiting — docs/USAGE.md §35) + background registry (v1.0.99 BG-REGISTRY: every initiated background work registers in <config-dir>/bg-registry.jsonl — open row per registration, close row per end, append-only; CLI bg-register/bg-close/bg-list [--active], automatic rows for unit-watch/ci-watch/--watch-pr and for every sweep tick — the crash traceability + the idle false-alarm read endpoint) + the update-validation sandbox (update-sandbox, v1.0.75: every ZCode update is validated FIRST in a fake-home rig — contract canary + turn-completion-rate against the staged candidate, GREEN/YELLOW/RED dossier, read-only vs the station — the constitution is docs/UPDATE-PROTOCOL.md)
contract_tests.py	Upstream contract canary — asserts every unpaid ZCode assumption against the live installation (see The contract canary)

The observatory cockpit for all of this lives in its own repo — see Pair it with the observatory.

The Narf-doorbell (--notify-parent + stream.py)

Children dispatched with --notify-parent <sess_id> ring the parent when they reach a terminal state. The parent wakes to a digest turn — status, token usage, result-file path, and the child's final response excerpt — persisted as a normal visible user message, so both the owner (transcript) and the model (next conversation's context) see it. Busy parents are detected via transcript activity (turn_usage rows only exist after turn completion) and the doorbell self-queues into stream-pending.json for the sweeper instead of racing the running turn. The sweep delivers oldest-first with separate rate budgets for bells and owner steers (owner input is never throttled by bell spam protection), a starvation guard for old entries, and a bell-ledger watchdog that re-queues and escalates any bell whose spawn was lost. The desktop-visibility mechanics of a delivered bell (the ~10-minute open-chat residency cycle, hydration on refocus) are dissected in docs/PROTOCOL.md — the observatory turns them into a per-session hydration prediction. The per-session bell contract (v1.0.56, RECONCILER). The bell contract is per SESSION, not per spawn: a create's --notify-parent persists the parent edge durably (stream-edges.json), and later steer/resume dispatches of the same child inherit the bell (--no-parent-bell opts out; provenance rides the result JSON and the lifecycle row as inherited-edge). The completion reconciler closes the residual class: a terminal dispatch row that never promised a bell whose lineage resolves a fresh parent edge gets ONE synthesized, receipt-protected bell (<dispatch-id>-reconciled) — idempotent, loop-guarded, quota deaths excepted; genuinely parentless children stay silent. Edge direction is a one-way contract (v1.0.75 REGRESSFOLD, the gen-4 night-watch cure): the edge recorder writes only the true direction (child = the driven session, parent = the ordered parent) and refuses the inverted twin — a mutual A→B + B→A pair is the echo ping-pong the 2026-10-17 genesis inversion fed on (47 mirror-served turns in 2.5 h); the surgical bell-parent seam repairs a legacy pair in one step, and the reconciler's master guard never serves an implicit lineage for a family head without a notify order of its own — a master's turn-ends reconcile nothing downstream. The busy-silent surface joins terminal rows + bell presence + lineage into completions-silent-state.json each tick (schema zcode/dispatch/silent-completions/v1) — display-only, the SILENCE-NEVER-KILLS invariant held structurally, the observatory B-side badges silent completions from it. The incident class (2026-10-14 morning standstill: a parented child's decisive steer-driven turn completed bell-less and the parent stood still 1h41m) is closed at all three layers. Quota deaths now also preserve the provider's reset moment (will reset at <ts>, provider-local UTC+8) into the death record and the revival state — the revival fires AT the reset, never on blind cadence. Details: docs/USAGE.md §8 + §18. Surgical reparent + succession hygiene (v1.0.75, REPARENT). Every succession used to leave the sons' father-bells ringing the RETIRING master (gen-4 and gen-5 both suffered it; the interim was genesis-record surgery — the product owns it now). dispatch.py --reparent <sess_id> (--to <sess_id> | --mute) --reason re-addresses the bell contract of a LIVING session — a CONTRACT EDIT with audit (override row + edge echo + the loud who/when/why mutation row; receipts and history stand), never a delivery hack, running turns untouched: the completion bell re-reads the contract at ring time, so a re-pointed session's bell rings the NEW parent (parent_source: surgical-override on the bell row) and a muted session's bell never fires at all (bell_suppressed on the dispatch row — no intent, no spawn, no queue entry). At the birth seam, a create declaring --succession-of <retiring> (+ --succession-reparent; config bell.succession_reparent: off | opt-in (default) | auto) re-points every child whose contract currently resolves to the retiring sid onto the newborn in ONE atomic audited batch — surgically-pointed children left alone, dead children skipped, the declaration always audited, a hygiene failure never disturbing the birth. The same core powers stream.py bell-parent (§27 of the usage doc). Details: docs/USAGE.md §8.

Details: docs/USAGE.md §8, event topics: docs/MQTT-TOPICS.md. Push (no polling): the stream.py events SSE bridge serves the same bus as zcode/events/v1-enveloped Server-Sent Events — docs/EVENTS.md.

Brain-children: delegation topology (--child-role brain, v0.3.6)

A brain dispatches two kinds of children: pinkies (workers — the default) and brain-children (delegations). A child launched with --child-role brain alongside --notify-parent is expected to run its own orchestra — spawn its own pinkies, recursively — instead of being a worker itself, and the machinery knows which kind it launched: the role rides the result JSON ("childRole": "brain"), the pinned static title ([🧠] project · first prompt line (tag) — pinky titles stay un-prefixed), the bell and lifecycle ledger records ("role": "brain"), and the doorbell prompt (a woken parent reads 🧠 brain-child sess_… returned instead of child sess_… returned, so a returning delegation is immediately distinguishable from a worker result — the reaper's presumed-dead alert uses the same distinction). The role survives the whole delivery chain: busy-parent self-queues, watchdog rescues and sweeper re-deliveries all carry it (env-marker relay ZCODE_DISPATCH_NOTIFY_ROLE). The observatory (v1.22.0+) renders the delegation topology from these records: 🧠 badges on brain-children and visually stronger brain-child edges in the lineage graph, separating the delegation backbone from the worker swarms.

CC bells: multi-parent fan-out (--notify-cc, v1.0.19)

One --notify-parent stays the single owner of a child's result — but trees and observers need more ears. --notify-cc sess_a,sess_b rings each listed session with a bell-only CC copy of the same doorbell digest when the child reaches a terminal state: recipient cwd/model auto-detected per recipient, one independent notification steer each (own systemd scope, own <dispatch-id>-cc-<recipient> bell id, kind:"cc" ledger record carrying the sender), CC doorbell wording that informs instead of parenting ("this is a bell-only CC copy from …, digest it only if your mission expects it"). --cc-on failure keeps observers on failed/timeout only. Busy recipients queue exactly like busy parents, delivery-once holds per recipient, and both the bell RATE budget and the token accounting count the fan-out against the sender (the recipient's budgets are untouched). The lifecycle MQTT payloads and retained state snapshots gain an optional cc: [<sids>] array so consumers can draw the CC edges. Observers on a parentless dispatch are legal: --notify-cc without --notify-parent, sender = the new session itself. Details: docs/USAGE.md §8.

Dispatch lifecycle ledger + silent-death reaper (v0.3.4)

Three dispatchers died young and silently in one day (2026-09-25 — crash, OOM-kill or SIGKILL, all with empty logs, noticed only because the Running table went empty). The lifecycle ledger makes that class of death visible: every dispatch writes a record to ~/.config/zcode-dispatch/stream-dispatches.json at START ({id, pid, cwd, model, prompt_file, out, started_at, notify_parent, role}), stamps session_id once the session materializes, static_title at the title-pin moments (v1.0.33), and a terminal {status, finished_at, exit_code} on every known exit path — the lifecycle is fired → pending → running → done/presumed-dead (the observatory shows the pending phase as its own strip).

A dispatcher whose process dies without a terminal update is a silent death; each stream.py sweep reaps those (pid forensics via /proc — zombie- and pid-reuse-aware), stamps the record presumed_dead_at, publishes zcode/dispatch/<id>/presumed-dead to the MQTT bus, and — the primary alert — rings the firing parent's 🔔 bell with the full trail (child cwd, result path, age), so the orchestrating brain can re-dispatch or investigate without any human in the loop. A desktop echo follows (respects the mute marker). When presumed-dead events repeat — 3 within 15 minutes, the same budget pattern as the bells — the system escalates to the owner desktop directly, and this escalation deliberately bypasses the notifications-mute marker (safety-critical: an orchestra losing workers silently must not be muted; the mute stays absolute for every other notification class). Env knobs: ZDO_REAP_GRACE_S, ZDO_REAP_ESCALATE_N, ZDO_REAP_ESCALATE_WINDOW_S. Details: docs/USAGE.md §8.

The weather cure: stream-drop close + auto-rerun (v1.0.75)

The 2026-10-23/24 network-storm census pinned ONE fleet-killing class: a mid-stream SSE socket drop (network_error/"terminated"/UND_ERR_SOCKET, classified retryable=false by the 3.16.3 turn-lifecycle path) leaves the turn permanently open — one final model.request.started fires ~20 ms after the drop chain and never reports; no turn.failed, no turn_usage row, no closing text (75/75 deterministic; the row is written by the turn-CLOSING bookkeeping, which never runs). The waiting dispatcher polled a row that never comes until the generic timeout, and orphaned sessions sat dead with their work unvisible. The cure has two halves:

• Clean close (the dispatcher's stream-drop watchdog). While a dispatcher waits on its turn, it incrementally harvests the CLI log's request-lifecycle chain (the same family the quota classifier reads). When the census signature holds — a retryable=false network stream failure immediately before the final model.request.started, no terminal request event and no turn closer after it, and ZDO_STREAM_DROP_SILENCE_S (default 90 s) of wall-clock silence — it writes the honest closer turn_usage row itself (status 'error', error_type 'stream-drop', the TRUE turn id + start moment + tool count harvested from the chain, idempotent per turn). The existing completion machinery then finishes the ordinary way: turn-done → result failed with error_class: stream-drop (result JSON, dispatch ledger, terminal MQTT payload) → the bell rings honestly — never the generic timeout lie ("may still be running"). A live retry ladder (retryable=true) always reports after its attempts and never classifies; a healthy long generation with an ancient retryable=false scar is excluded by the cause-adjacency window.
• Auto-rerun, the owner-click turn-retry productized (the sweep's silent_turn_death_pass). For sessions nobody waits on (terminal dispatch-ledger rows inside the lookback), the sweep reads transcript truth read-only: the census shape is a quiet tool tail (last part older than ZDO_STREAMDROP_QUIET_S, default 6 min) with tool parts, no closing assistant text after the last tool part, and no closer row for the window. On detection it writes the same closer row and queues ONE continue-nudge as an ordinary steer (never a re-send of the original words — the receipts/ONCE machinery guards it like any owner steer; it rides the same-tick delivery loop under the serialization/FIFO/receipts laws). ONCE per open window (the anchor is spent after one audit record in stream-turnretries.json), bounded per session (ZDO_STREAMDROP_MAX_PER_SESSION, default 2) and per tick — a spent budget gives up loudly (event + audit, closer row, no nudge). Live sessions are excluded by construction: a live in-flight marker, a running turn row, a non-terminal dispatch row, or a retired/archived target all defer; owner-interactive sessions outside the dispatch ledger are deliberately untouched (the observatory's retry affordance remains theirs).
• Weather made visible. Today's counts — closed (dispatcher-side closes), detected, rerun — publish retained on change as zcode/dispatch/stream-drops (the bells-unconfirmed rollup pattern; state in stream-drop-state.json), the tick summary carries streamdrop(…) counters, and stream.py status prints the day line. Env knobs: ZDO_STREAM_DROP_SILENCE_S, ZDO_STREAMDROP_QUIET_S, ZDO_STREAMDROP_LOOKBACK_S, ZDO_STREAMDROP_MAX_PER_SESSION, ZDO_STREAMDROP_PER_TICK.

Orchestration-chain hardening (v1.0.1)

die Orchestrierungskette darf keinesfalls unterbrochen werden

The triple crisis of 2026-09-25/26 (broker exited with no restart policy and sat dead for ~6 h; a notification steer spawned with plain setsid died mid-start when the spawner's exec cgroup was torn down at turn end; the observatory crashed with nothing to restart it) mapped the chain's single points of failure. The chain is:

bells (doorbell spawns) → MQTT broker → observatory/owner (consumers)

v1.0.1 puts a safety net under each link:

• Broker: restart policy + watchdog. Recreate the station's broker with --restart=always so podman itself revives it on any exit (verified live: a crash-killed mosquitto returns in ~4 s). On top, stream.py mqtt-watchdog runs one sub-second TCP probe per tick, fires the recovery ladder on failure (the podman ladder, v1.0.3: start, then restart for a wedged host port proxy, then a full rm+run recreate carrying the restart policy — each rung matches a failure mode observed live; every rung is executed one at a time, scope-isolated via systemd-run --user --scope into a zcode-mqtt-recovery-* unit and verified by a 2 s TCP port probe — exit codes are not trusted (podman start exits 0 on a running-but-unbound container), and only a reachable port ends the ladder; the scope isolation keeps the podman helpers (slirp4netns/rootlessport/conmon) out of the watchdog oneshot's cgroup, whose teardown would SIGKILL them ~90 s later and delete the bind the recovery just created — the v1.0.2 defect behind 6 outages in one night; ZDO_MQTT_START_CMD replaces the ladder with one custom command, ZCODE_MQTT_CONTAINER renames the container), sends a CRITICAL desktop alert that bypasses the notifications mute (an outage deafens every downstream alert channel — same safety class as the reaper escalation), publishes zcode/infra/mqtt-restarted once the broker answers again, and logs the outage duration to ~/.config/zcode-dispatch/mqtt-watchdog.json. Every stream.py sweep tick embeds one pass (15 s ceiling); for the fierce cadence run a dedicated fast timer (station: 5 s — see docs/USAGE.md §14). Env knobs: ZDO_MQTT_STREAK (failed ticks before recovery, default 1), ZDO_MQTT_PROBE_S, ZDO_MQTT_RECOVERY_RETRY_S.
• Notification steers: their own systemd scope. --notify-parent bells now spawn via systemd-run --user --unit=zcode-notify-<hash> — a scope no parent-cgroup teardown can reach (setsid detaches the session, not the cgroup, which is exactly how the crisis bell died). Fallback on non-systemd stations: the legacy setsid spawn, warned about in the result JSON (notifySpawnWarning) and stamped in the bell ledger (spawn_mode). Belt and suspenders: a notification dispatcher that finds itself spawned outside systemd (marker ZCODE_DISPATCH_NOTIFY_SPAWNED unset — an older dispatcher or a foreign spawner) re-execs itself into a zcode-notify-* scope at startup before taking any locks. Sweep- and observatory-managed children carry managed-parent instead — their spawners wait on the exit code, and a re-exec would fake a delivery record.
• Canary: the chain itself is an assumption. The contract canary's infra group performs a real MQTT connect+publish roundtrip on every run — a red infra.mqtt-broker means the watchdog failed, not just the broker, and the observatory's contract banner doubles as the infrastructure health indicator.
• Observatory crash recovery ships in the observatory repo (v1.25.0): a Restart=on-failure systemd unit, deliberately not auto-enabled.

The contract canary: upstream-change alarm (v1.0.0)

Everything above rides on unpaid assumptions about ZCode internals — the session-DB schema and its JSON payload paths, the app-server protocol shapes, product facts reverse-engineered from the installed bundle. An upstream update can break any of them silently. The contract canary (contract_tests.py) turns that risk into an alarm: one fast, read-only command that probes the live installation and asserts every assumption, one PASS/FAIL line each, with the product feature it breaks named on every FAIL — so triage is instant.

make contract                # or: python3 contract_tests.py
make contract-deep           # additionally runs ONE real model turn
stream.py contract [--deep]  # same, via the stream tool

Five groups, ~41 assumptions: env (binary answers, DB/credential/ provider-config paths, CLI-log registry revision), infra (the MQTT broker connect+publish roundtrip — the orchestration chain's spine; v1.0.1), db (pinned columns for session/message/part/turn_usage/model_usage/tasks-index plus the live JSON paths — semantics, modelSelection, contextSnapshot.envInfo.cwd, part.$.text, part.$.tool), bundle (mechanism strings in the installed zcode.cjs: question auto-resolution, runtime-headers handshake, the -32010 wording, the residency deactivation), and proto — a live app-server probe on ONE self-cleaned scratch session (archived afterwards): startup env, entitlement push, session/create, runtime-preferences request, strict send schema, content ACK, runtime-headers handshake, the same-process -32010 guard, session/usage. --deep adds a real scratch dispatch and proves the send-written semantics, the transcript trace, the title pin and the turn-window anchors end to end (the fast probe makes NO model requests — its probe turn declines auth and errors out instead).

The sticky runtime snapshot (v1.0.75 JUGGLEDEFENSE): the app-server stamps its provider-registry revision from the resolved path of its zcode-builtin.json (zcode-builtin:<int>:<sha256(path)>), so an update window splits instances across eras and a log-harvested entitlement push coin-flips — the 3.16.3 flip's 50–90k turn-death wave. The dispatcher now keeps a content-addressed, immutable copy of the newest runtime snapshot under the config dir (two-observation debounce, never mutated mid-flight), pins spawns to it via the ZCODE_BUILTIN_PROVIDER_CONFIG_FILE pair, and re-anchors the push onto the copy's computable path-hash revision; the registry-race self-heal rides only the child's own ready line (path-hash proof) and re-targets at most once. Fail-open: any snapshot problem falls back to the pre-75 behavior. See docs/USAGE.md §32.

Automatic post-update trigger

The update moment is exactly when breakage lands, so the canary fires there: every stream.py sweep tick (the existing timer lane) stats the zcode-cli wrapper exec-chain artifacts (~/.local/bin/zcode-cli → /usr/bin/zcode-cli → the zcode.cjs bundle — cheap stat, no subprocess), compares them against the stamp file ~/.config/zcode-dispatch/contract-cli-version.stamp, and a missing stamp (baseline) or any version/artifact change fires the suite immediately. The stamp is (re)written only after a green run — a red canary keeps retrying every sweep until it is fixed or understood.

Since v1.0.57 the fire itself is transport-resilient (docs/USAGE.md § 23): a declared maintenance window (the ops contract in docs/CONTRACTS.md — freezes announced by Astra/Root become one JSON write) defers the canary with a visible reason and re-fires on its own after the settle margin; a pre-fire health probe waits out a flapping CLI instead of firing onto the blade; the suite's own probe legs ride bounded retry, and a run killed mid-way resumes from its per-group checkpoint instead of redoing everything (replayed checks are marked as such in the report).

Alert channels

• MQTT: zcode/contract/broken (payload carries the per-assumption failed list with its breaks-feature annotations and the version-drift info) / zcode/contract/ok — see docs/MQTT-TOPICS.md.
• Desktop: a notification when the canary transitions into broken (and a green-again note when it recovers) — mute-RESPECTING (maintenance-grade; only the reaper's silent-death escalation and the broker watchdog's CRITICAL outage alert ever bypass the mute).
• Observatory: the cockpit reads the suite's JSON report (~/.config/zcode-dispatch/contract-status.json) in its existing poll and shows a contract banner — subtle amber when broken (click through to the failed assumptions with their breaks-feature notes and the old → new version drift), green-tinted when healthy. No report file = feature too new or never run: the banner simply stays away.

The first live runs (2026-09-25) already paid for the whole design: the suite caught two of its own probe assumptions being wrong about the product (session rows materialize at the first send, not at create ACK; the -32010 window closes within seconds on an errored turn) — both now encoded as corrected facts in docs/PROTOCOL.md.

Pending AskUserQuestions: park + external answering (v0.3.0)

AskUserQuestion tool calls (interaction/requestUserInput) no longer rely solely on the app-server's auto-resolution. The dispatcher parks every incoming question into the result JSON (pendingQuestion) and into ~/.config/zcode-dispatch/questions/<session>.pending.json; anything — today zcode-dispatch-observatory — answers by writing <session>.answer.json ({"requestId", "action", "answers": {<question>: <answer>}}), which the dispatcher forwards to the app-server within ~1 s. The turn then continues exactly as if the owner had clicked the option.

With ZCODE_DISPATCH_ANSWERABLE=1 the runtime preferences disable the app-server's 5-minute empty-answer auto-resolution, so a parked question waits for the external answer indefinitely; without it the current behavior holds (auto-resolve empty at 300 s — an external answer still wins inside the window). Desktop-driven sessions cannot be answered through this channel (their interactions live inside the desktop app's private app-server) — the observatory shows those questions read-only. Full protocol forensics: docs/PROTOCOL.md § AskUserQuestion.

Tool approvals through the same side-channel (v1.0.75, APPROVE)

Tool-level permission requests (interaction/requestPermission — the attempt2 class: a find/sed approval hung a dispatched session for hours with nothing answerable on the deck) now flow through the SAME side-channel family. The dispatcher parks every incoming request as questions/<session>.pending.json with kind: "tool-approval", a command preview, risk level and grant/deny semantics (the result JSON mirrors it as pendingToolApproval). An answer file {"requestId", "action": "grant"|"deny", "reason"?} is forwarded as the strict protocol decision within ~1 s — grant releases the turn, deny terminalizes honestly (exit 4); a malformed action DENIES (a permission gate never fails open). There is no server-side auto-resolution for permission requests, so a parked approval waits for the external answer indefinitely. Under the full-access decree (yolo default) no request is ever made — these events are the gated-exception surface (--mode build|edit|plan) made answerable. The sweep additionally surfaces approval-stuck dispatched sessions from the session DB (tool_usage.approval_status='requested') for pre-v1.0.75 drivers — honestly NOT file-answerable there (no wire requestId; the card names the escapes). Full wire/file contract: docs/PROTOCOL.md § Tool approvals.

Long-running tasks from agent sessions — the detach law

The law: a spawning harness tears down its exec cgroup when YOUR turn ends — every plain backgrounded child (setsid nohup … &, &, background tool tasks) dies with it, typically within minutes, no matter how healthy it looked. Observed repeatedly on-station (coordinated kills at turn end, empty logs = SIGKILL on the cgroup); a 2026-10-14 gate "death" first read as a transport failure and was actually this class — the transport never failed. Long-running work launched from an agent session must run in its own scope:

# the built-in way for dispatches (v1.0.34) — fire-and-forget:
zcode-dispatch --systemd --cwd <worktree> --model GLM-5.3 \
  --prompt-file <task.md> --out <run.json>

# anything else (gates, e2e stacks, watchers):
systemd-run --user --unit <name> --collect bash -c '…'
systemctl --user is-active <name>   # verify it holds past turn end

--systemd re-execs the whole dispatch into a zdispatch-* user scope before any state mutation (full mechanics: USAGE § 11). For manual systemd-run invocations remember: the child does NOT inherit the caller's env (forward ZCODE_*/ZDO_*/PATH explicitly, add --same-dir for relative argv) and argv must carry zero braced parameter forms (${10} expands to empty — compose flag pairs caller-side).

Family distinction (do not confuse the two): this is harness-side turn-end teardown — the harness kills everything in the exec cgroup when the TURN ends, and only a scope survives it. The other family is watcher time-blades — a supervisor intentionally killing a child that exceeded its time budget (the sweep's delivery timeout, the canary's CONTRACT_TIMEOUT_S). Same symptom (a dead child), opposite medicine: teardown is survived by detaching; a time-blade is answered by the gate's own bounded-retry/checkpoint machinery (docs/USAGE.md § 23) — never by detaching away from the watcher.

The app-driver channel (v1.0.80→81, S1)

app_driver.py is a SECOND delivery channel beside the CLI path: it launches and drives an OWN ZCode desktop-app instance (own userData → own SingletonLock, own Xvfb display, private D-Bus, own CDP port) and rides everything first class — the OAuth store, the armed stream watchdog, the reconnect ladder, the self-provisioned runtime. Input seam is the UI composer over CDP (own instance only — the cockpit law ENFORCED: a /proc-proven ownership chain — listener pid environ nonce + seams + singleton + marker — gates every attach, and owner-port neighborhoods (±100) are refused before anything runs; the owner's app exposes no external seams anyway); output truth is the shared session DB (turn_usage completion + tokens), so app-driven turns are observatory-visible without new ledgers. The 2026-10-26 APP-BRUECKE round proved the path live end-to-end. Today it is a standalone channel (plan/ensure/drive/status/stop), not a CLI replacement — see docs/USAGE.md §33 and the seam map in docs/PROTOCOL.md.

Documentation

• docs/USAGE.md — dispatcher guide: flags, exit codes, result schema, detached launches, failure recovery, gate transport resilience
• docs/PROTOCOL.md — app-server protocol findings (entitlement mirroring, runtime headers, visibility)
• docs/CLI-REFERENCE.md — zcode-cli terminal reference: headless mode, session store, skills/plugins/MCP configuration
• docs/PROMPT-APPENDIX.md — paste-ready prompt appendix for orchestrating agents (Codex/Astra, Claude Code, scripts)
• docs/RELEASE.md — release process: monotonic versioning, deterministic git archive tarball, laboratory asset upload recipe
• docs/EVENTS.md — push transports + the zcode/events/v1 envelope: the consumer contract for stream.py events (SSE wire format, topic→type mapping, replay/reconnect, CORS) and stream.py webhooks (signed POSTs, registration schema, retry/dead-letter semantics, HMAC recipe, systemd recipes)
• docs/BUDGETS.md — quota & usage accounting + budget guardrails: the usage ledger (lane-keyed token accounting), admission-only soft/hard limits (exit 7, --budget-override), retained MQTT usage rollups (legacy + fleet-scoped), the --usage-summary command, and how consumers (the observatory's quota/cost lane) wire on
• docs/QUOTA.md — the Z.AI quota monitor API contract (v1.0.49): the key's resolution chain (env > config > key file), --quota output schema (human + JSON), --set-quota-key absorption semantics (validate-before-persist, atomic 0600 write, masked confirm) and the binding key-hygiene rules
• docs/MQTT-TOPICS.md — the full MQTT topic layout: lifecycle/infra/usage topics, the retained zcode/state/v1 session snapshots (archived tombstones), the fleet namespace zcode/fleet/<station-id>/… (station-id derivation, legacy→fleet mapping, deprecation note) and the broker-restart repopulation runbook
• docs/ROADMAP.md — roadmap from v1.0.5 onwards: push-based delivery (P0), budgets + retained snapshots (P1), CC bells, canary + packaging hardening
• docs/CONTRACTS.md — machine-readable contracts this repo publishes for ops/external tools: the maintenance-window.json freeze-window format (schema, reader semantics, settle/pre-grace rules)

Security

The plan API key is decrypted in dispatcher memory (AES-256-GCM, ZCode's own credential-store scheme) and passed exclusively over the local stdio pipe to the official zcode-cli binary. Keys never appear in argv, logs, results, or evidence files. Header requests for foreign provider/model contexts are declined. No global ZCode settings are modified.

Status

Battle-tested on ZCode Desktop 3.14.0 / CLI 0.16.9 (Linux x64). See docs/PROTOCOL.md for the protocol findings and known upstream limitations.

MIT licensed. Built in the OSS Singularity org — source of truth lives on oss-oo.io.

Roadmap

The checklist below is the shipped history in highlights — not every round gets a line (hotfix, packaging, and validation rounds fold into the features they hardened); the release tags carry the full sequence. Future work is tracked in docs/ROADMAP.md — from v1.0.5 onwards: push-based delivery (P0), budget guardrails and retained MQTT snapshots (P1), multi-parent CC bells and canary/packaging hardening (P2/P3), plus the upstream-blocked items.

• Static pinned session titles (start notify = sidebar = completion notify)
• Click-through desktop notifications (STARTED / COMPLETED / FAILED / TIMEOUT)
• Task-index seeding for desktop visibility (By project)
• Social preview typography (overlay rendered from social-preview-src/)
• Lint-clean main (Super-Linter: pylint/ruff/markdown)
• Per-repo --setup wizard (auto-detect provider, models, credentials)
• Pending-question side-channel (park + answer files, ZCODE_DISPATCH_ANSWERABLE=1)
• Tool-approval side-channel (kind="tool-approval" park/grant/deny + the sweep's approval-stuck DB net — v1.0.75)
• Release install flow (install.sh + release tarball, update-in-place)
• Narf-doorbell hardening (digest payload, transcript-activity busy guard, queue-on-busy — notifications never dropped)
• MQTT event bus (podman mosquitto, zcode/session/+/+ topics, stream.py listen — v0.2.x)
• Bell reliability: rate-budget separation, anti-starvation, bell ledger + watchdog (v0.3.1)
• Ghost-delivery fix: send-trace completion invariant — exit 0 only when the prompt text verifiably reached the transcript; swallowed sends re-queue instead of faking delivery (v0.3.3)
• Dispatch lifecycle ledger + silent-death reaper — every dispatch ledgered fired → pending → running → done; dead-without-terminal dispatches presumed-dead, parent belled, repeating deaths escalate to the owner past the mute (v0.3.4)
• Child-role flag: --child-role {pinky,brain} — brain-children are delegations (sub-brains running their own orchestras); the role rides the result JSON, the [🧠] title prefix, both ledgers, the doorbell wording and the whole re-delivery chain, feeding the observatory's delegation topology (v0.3.6)
• Upstream contract canary — every unpaid ZCode assumption asserted against the live installation (env/db/bundle/proto groups + a real-turn deep mode), auto-fired on zcode-cli version changes by the sweep's version watch, alerting via MQTT, desktop and the observatory banner (v1.0.0)
• Orchestration-chain hardening — MQTT broker restart policy + fierce watchdog (recovery, CRITICAL mute-bypassing alert, outage log, zcode/infra/mqtt-restarted), notification steers spawned into their own systemd scopes (with a startup self-check re-exec and a bell-ledger spawn-mode audit trail), the canary's infra group asserting the chain itself (v1.0.1)
• Delivery-once hardening — every successful/dropped sweep delivery persists to the history file AT ONCE (not at sweep end) and each item re-checks the freshly-read history before spawning: a turn-end fast-lane delivery that lands mid-sweep can no longer be re-delivered by the sweep's stale in-memory view (forensic: one steer drove two turns 17 s apart, 2026-09-25 23:40)
• Notification clicks land in the observatory — notify_click.sh defaults to xdg-open $ZDO_OBSERVATORY_URL/?session=<id> (port-probed, zcode:// fallback), making every bell one click from the conversation
• Watchdog scope isolation + port-probe ladder — every recovery rung runs systemd-run --user --scope zcode-mqtt-recovery-* (the podman helpers must outlive the oneshot service's cgroup teardown) and is verified by a TCP port probe, never by its exit code (podman start exits 0 on a running-but-unbound container); timer service units get TimeoutStopSec=5 (v1.0.3)
• SSE push transport (P0 first half) — stream.py events: one zcode/# MQTT subscription fanned out to N EventSource consumers on GET /events, wrapped in the versioned zcode/events/v1 envelope (topic→type mapping, digest, Last-Event-ID replay, keepalives, per-consumer drop notices, CORS); the consumer contract is docs/EVENTS.md (v1.0.8)
• Outbound webhooks V1 (P0 second half) — stream.py webhooks: one signed zcode/events/v1 POST per registered consumer per bus event (HMAC, retries with backoff, dead-letter JSONL, MQTT notices, self-notice loop prevention, kill-switch); see docs/EVENTS.md (v1.0.9)
• Residency findings (10-minute open-chat cycle → observatory hydration prediction)
• Cross-leverage story: observatory pairing (steer UI, answers, archive, lineage)
• Social preview final polish (six-band card, repo footer, 1 MiB-optimized render)
• Quota & usage accounting + budget guardrails (roadmap P1) — rolling-window lane/tree token accounting persisted ledger-style, soft warning bell + hard refusal before any spawn (exit 7 budget-refused, audited --budget-override bypass), station-day/lane usage rollups retained via MQTT; see docs/BUDGETS.md (v1.0.16)
• Retained state snapshots + station-fleet topics — zcode/session/<id>/state retained last-known-state (archived tombstones prune), zcode/fleet/<station-id>/… dual-published mirrors, station-day/lane usage rollups scoped per station, lane-prune refresh, broker-restart repopulation drill (v1.0.18)
• Multi-parent / CC bells (roadmap P2) — --notify-cc sid[,sid…] + --cc-on {all,failure}: bell-only CC copies of the doorbell digest to observer sessions (per-recipient steers, kind:"cc" ledger records with sender, distinct cc bell ids, sender-keyed rate buckets + sender-lane token attribution, additive cc MQTT fields, full validation set), plus the valr F-1/F-2/F-3 fold-ins (publish_dual ASCII byte-compat, suite-guarded archive tombstone, behavioral bridge mirror-skip) (v1.0.19)
• Install self-verification + gate isolation (roadmap P3) — install.sh --check: the station acceptance triple as a first-class command (payload byte-identity vs a directory / git ref / origin-main, FILES-list packaging guard, symlink and exec-bit checks, exit 0/1/2) with read-only --live unit reports; make check now pins its dispatcher config dir to a repo-local scratch so station gate runs never append to the real ledgers (VEE6-F3) (v1.0.21)
• Incident snapshots + canary drift alarm (roadmap P4b) — stream.py snapshot freezes the five rotation-capped ledgers under incidents/<utc-ms>/ (byte-faithful or --window-filtered, manifest with counts/reason/version, ZDO_SNAPSHOT_KEEP retention) and the dispatcher's server-side-cancel detection auto-freezes one dir per ZDO_SNAPSHOT_MIN_INTERVAL_S (the 2026-10-01 race forensics lesson: rotation ate the evidence); stream.py canary-report runs the canary's read-only probe groups (contract_tests.py --probe) and alarms on group-outcome DRIFT vs canary-baseline.json — exit 0 quiet / 1 drift + zcode/infra/canary-drift dual-published non-retained / 2 probe-infrastructure error, --rebaseline remedy — with an hourly systemd timer shipped DISABLED by default (the bridge precedent; runbook in docs/USAGE.md §16–17) (v1.0.24)
• Lineage-aware desktop-echo gating — notifications.gate / ZDO_NOTIFY_GATE (all | smart | muted) gates BOTH echo sites (STARTED + terminal) so the desktop only hears sessions in the owner's direct care: steers echo iff the target is not chain-internal, creates echo unless chain pinkies (brain children always), fail-safe to echo on unreadable ledgers, mute FILE above every mode. Default changed to smart on upgrade — sub-brain chain noise disappears; gate: all restores the old behavior (v1.0.27, docs/USAGE.md §4)

Feel the data flow.

Credits

• Social preview artwork by Luna ✨ — the moment in between: a thought is sent out, flows across the bridge as work, and returns as a finished result.
• Protocol findings documented from ZCode Desktop 3.14.0 / CLI 0.16.9 — upstream ticket ZCT-21013117278467761684 (timeline visibility resolved upstream in 3.15.1, owner-verified 2026-10-12).
• Background registry (v1.0.99 BG-REGISTRY) — every initiated background work registers in <config-dir>/bg-registry.jsonl: one append-only OPEN row per registration (id, purpose, scope, owner{session,name}, deadline_ts, result_path, started_ts) and one CLOSE row (id, ended_ts, outcome, exit) written under a sidecar flock; a crashed run leaves its open row behind — the registry IS the what-ran/what-should-have-run traceability. Seams: stream.py bg-register/bg-close manual CLI (refusals exit 2), automatic rows for unit-watch/ci-watch/--watch-pr arming (terminal rung/expired closes them; refresh never double-registers), every sweep tick registers and closes itself; stream.py bg-list [--active] is the read endpoint (derived age_s/overdue, torn lines skipped loudly) — the idle false-alarm killer the observatory's P2 display builds on (operator commission 2026-10-28 21:29)