Bell reliability: rate-budget separation, anti-starvation, bell ledger + watchdog (v0.3.1)

ClaudiuSchuster commented

2026-09-24 22:35:43 +00:00

Owner

Fixes the owner-reported chain break (2026-09-25): a validation bell sat undelivered for ~2 h because the shared per-parent rate limit (3/15 min) was consumed by the owner’s own queued steer deliveries while the parent was busy in long owner conversations.

Rate-budget separation — every queue/history entry gains a kind (bell vs steer, legacy classified by event="steer"); budgets are per kind per parent / 15 min: bells 3 (anti-spam), steers ZDO_STEER_RATE_LIMIT (default 20). Owner input is never throttled by bell spam protection.
Queue fairness + anti-starvation — the sweep delivers OLDEST entries first; anything queued longer than ZDO_STARVATION_S (default 600) delivers regardless of the budget, and every guard fire is logged.
Bell ledger — notify_parent() appends a spawn-time record to stream-bells.json {child, parent, event, spawned_at, result_out, …} (deduped per child+event, last 200); delivery stamps delivered_at — the authoritative bell audit trail.
Watchdog — each sweep rescues ledger bells that are neither queued nor delivered past ZDO_BELL_WATCHDOG_S (default 300): re-queued with original age (budget bypass), escalated via MQTT zcode/session/<parent>/bell-stuck {child, age, action} + one-shot desktop notification (“🔔 WATCHDOG: undelivered bell re-queued”); after 3 rescues a record is marked lost, never re-queued.
Offline functional test (stream_functional_test.py, wired into make functional): replays the exact incident (3 steer deliveries in window + queued bell → delivers via starvation guard / kind separation), proves oldest-first fairness, the intact bell throttle, the lost-spawn watchdog (requeue + escalation + ledger reconcile) and the spawn-time ledger + busy self-queue.

Docs: USAGE.md §8 (budgets, guard, ledger, watchdog, env vars), MQTT-TOPICS.md (bell-stuck topic), README, .zcode/AGENTS.md.

Fixes the owner-reported chain break (2026-09-25): a validation bell sat undelivered for ~2 h because the shared per-parent rate limit (3/15 min) was consumed by the owner’s own queued steer deliveries while the parent was busy in long owner conversations. - **Rate-budget separation** — every queue/history entry gains a kind (`bell` vs `steer`, legacy classified by `event="steer"`); budgets are per kind per parent / 15 min: bells 3 (anti-spam), steers `ZDO_STEER_RATE_LIMIT` (default 20). Owner input is never throttled by bell spam protection. - **Queue fairness + anti-starvation** — the sweep delivers OLDEST entries first; anything queued longer than `ZDO_STARVATION_S` (default 600) delivers regardless of the budget, and every guard fire is logged. - **Bell ledger** — `notify_parent()` appends a spawn-time record to `stream-bells.json` `{child, parent, event, spawned_at, result_out, …}` (deduped per child+event, last 200); delivery stamps `delivered_at` — the authoritative bell audit trail. - **Watchdog** — each sweep rescues ledger bells that are neither queued nor delivered past `ZDO_BELL_WATCHDOG_S` (default 300): re-queued with original age (budget bypass), escalated via MQTT `zcode/session/<parent>/bell-stuck` `{child, age, action}` + one-shot desktop notification (“🔔 WATCHDOG: undelivered bell re-queued”); after 3 rescues a record is marked lost, never re-queued. - **Offline functional test** (`stream_functional_test.py`, wired into `make functional`): replays the exact incident (3 steer deliveries in window + queued bell → delivers via starvation guard / kind separation), proves oldest-first fairness, the intact bell throttle, the lost-spawn watchdog (requeue + escalation + ledger reconcile) and the spawn-time ledger + busy self-queue. Docs: USAGE.md §8 (budgets, guard, ledger, watchdog, env vars), MQTT-TOPICS.md (bell-stuck topic), README, .zcode/AGENTS.md.

ClaudiuSchuster added 1 commit

2026-09-24 22:35:44 +00:00

Bell reliability: rate-budget separation, anti-starvation, bell ledger + watchdog

Checks / Functional checks (pull_request) Failing after 2m13s

Details Explain this check

5af26cb376

Fixes the owner-reported chain break (2026-09-25): a validation bell sat
undelivered for ~2 h because the shared per-parent rate limit (3/15 min)
was consumed by the owner's own queued steer deliveries while the parent
was busy in long owner conversations.

- Kind on every queue/history entry (bell vs steer, event="steer"
  fallback for legacy records); rate budgets are per kind per parent per
  15 min: bells 3 (anti-spam), steers ZDO_STEER_RATE_LIMIT (default 20) —
  owner input is never throttled by bell spam protection.
- Sweep delivers OLDEST queue entries first; a starvation guard
  (ZDO_STARVATION_S, default 600) delivers anything older regardless of
  the budget and logs every fire.
- Bell ledger: notify_parent() appends a spawn-time record to
  stream-bells.json {child, parent, event, spawned_at, result_out, …}
  (deduped per child+event, last 200); delivery stamps delivered_at —
  the authoritative bell audit trail.
- Watchdog: each sweep rescues ledger bells neither queued nor delivered
  past ZDO_BELL_WATCHDOG_S (default 300) — re-queued with original age
  (budget bypass), escalated via MQTT zcode/session/<parent>/bell-stuck
  {child, age, action} plus a one-shot desktop notification; after 3
  rescues a record is marked lost, never re-queued.
- stream_functional_test.py (wired into make functional): offline replay
  of the exact incident plus lost-spawn watchdog, fairness, throttle and
  spawn-time ledger legs against the real sweep code.

Docs: USAGE.md §8 (budgets, guard, ledger, watchdog, env vars),
MQTT-TOPICS.md (bell-stuck), README, .zcode/AGENTS.md.

ClaudiuSchuster added 1 commit

2026-09-24 22:52:43 +00:00

Sweep survives delivery timeouts; test watcher re-parses until turn-running

Checks / Functional checks (pull_request) Successful in 58s

Details Explain this check

07eec45424

CI (forge run 42) exposed two flaws:

- stream.py: a delivery exceeding ZDO_SWEEP_DELIVERY_TIMEOUT_S raised an
  uncaught TimeoutExpired that killed the whole sweep tick (pre-existing
  since the #18 timeout). A timed-out delivery now re-queues like the
  exit-3/5 busy paths — the wake turn may still be running server-side,
  and delivery dedupe prevents double bells.
- stream_functional_test.py: the turn-completion watcher marked a result
  file done on first parse, so on a slow runner it consumed the
  incremental file in its 'starting' state and the delivery hung until
  the sweep timeout. Files are now re-read every tick until they say
  turn-running, and sqlite hiccups retry instead of killing the thread.

Verified: 3x green locally, green in python:3.12-slim at 0.4 CPU,
make functional + Super-Linter slim v8.7.0 EXIT 0.

ClaudiuSchuster merged commit 062462bc7c into main

2026-09-24 22:54:07 +00:00

ClaudiuSchuster referenced this pull request from a commit

2026-09-24 22:54:09 +00:00

Bell reliability: rate-budget separation, anti-starvation, bell ledger + watchdog (v0.3.1) (#21)

Rows
Columns

Bell reliability: rate-budget separation, anti-starvation, bell ledger + watchdog (v0.3.1) #21