laboratory/docs/WATCHDOG.md

Independent watchdog

The watchdog package runs on a separate observer. It preserves the eight public probes and their incident deduplication keys. Shadow mode records observations without notification credentials. Active mode requires a confirmed cutover baseline and evidence that owner notifications survive a Laboratorium outage.

This source change does not deploy a service or change the station timer. The example configuration deliberately blocks activation: its ingest acknowledgement contract and independent notification evidence are unset. Existing station marker files do not prove that an alert or email was accepted, because the old script changed them regardless of delivery.

Preserved probe contract

Deduplication key	Exact alert title	Probe and expected result
oss-oo-probe-landing	oss-oo.io probe landing	/about/, HTTP 200
oss-oo-probe-web-login	oss-oo.io probe web login	/user/login, HTTP 200
oss-oo-probe-web-healthz	oss-oo.io probe healthz	/api/healthz, HTTP 200
oss-oo-probe-api	oss-oo.io probe api	/api/v1/version, HTTP 200
oss-oo-probe-api-mcp	oss-oo.io probe api mcp	/mcp/healthz, HTTP 200
oss-oo-probe-oci	oss-oo.io probe oci	/v2/, HTTP 401
oss-oo-probe-git	oss-oo.io probe git hosting	/OSS-OO/laboratory.git/info/refs?service=git-upload-pack, HTTP 200
oss-oo-probe-events	oss-oo.io probe event bus	events.oss-oo.io, verified TLS WebSocket /mqtt, authenticated MQTT 3.1.1 CONNACK accepted

HTTP probes do not follow redirects or use environment proxies. An OCI 401 is healthy only for its own probe; other checks still require 200. Missing MQTT credentials are a configuration failure, not a skipped eighth probe. MQTT has no fixed-IP DNS fallback. Each connection uses a different client ID, so shadow cannot disconnect the station's probe at the broker.

The existing metadata.source identifier remains station-watchdog for route compatibility. metadata.observer identifies the new owner. The identifier is a compatibility field, not a claim that the observer runs on the station. Verify any provider filters before cutover.

Independent delivery is a prerequisite

The old ops-mail helper enters a container on Laboratorium and submits SMTP to its local relay, which forwards through Graph. The observer must not call this helper through SSH, use the station's administration key, or copy the broad mail application credential.

The implemented channel is direct incident-source HTTPS ingest with separately verified platform notifications. It can replace the old parallel ops-mail only after the operator verifies equivalent DOWN and recovery delivery to the existing recipients, exact title/status routing, status-page behavior, and retry deduplication. A successful ingest response alone proves none of these outcomes. Until that evidence exists, leave the station active and use shadow only.

If the existing platform cannot provide equivalent notifications, cutover stays blocked. An independently scoped mail delivery channel and separate per-channel receipts would be additional reviewed work. No independent SMTP endpoint or platform notification capability is assumed by this package. Do not remove the parallel ops-mail requirement just to pass the activation check.

The operator can inspect the existing alert-source setup/request history, route, notification workflows and recipient configuration without sending a test event. Recover the actual success HTTP status/body rule and the current status of all eight keys there. If read-only evidence cannot establish recipient equivalence and actual delivery, the owner must choose and authorize either a bounded notification drill or a separately scoped independent mail channel. Do not manufacture the evidence file from the fact that credentials exist.

The official incident alert setup and routing documentation describe ingest, escalation and incident creation as distinct configuration steps. The official HTTP integration example documents the bearer-authenticated JSON fields. These references do not establish the installed account's response schema, plan capabilities, recipients or delivery outcome.

Files and privileges

Install the two Python modules as root-owned, read-only files under /opt/oss-watchdog/. The service account needs no shell administration key, sudo, Docker socket or access to monitored-host files.

Generic path	Purpose
/etc/oss-watchdog/config.json	Nonsecret configuration, based on watchdog/config.example.json
/etc/oss-watchdog/mqtt.json	Private {"password": "<credential>"}
/etc/oss-watchdog/alert.json	Private {"url": "<ingest HTTPS URL>", "token": "<ingest token>"}
/etc/oss-watchdog/notification-verification.json	Private operator evidence attestation
/var/lib/oss-oo-watchdog/active-state.json	Durable active observations, pending transitions and acknowledgements
/var/lib/oss-oo-watchdog-shadow/shadow.json	Shadow observations; never an acknowledged baseline

Secret and evidence inputs must be regular files, single-linked and owner-only readable (0600 or 0400 at the source). systemd LoadCredential makes them available as mqtt.json, alert.json and notification.json in its private credential directory. The credential-only reader supports the observed systemd255 root:root 0440 files and 0550 directory: exactly root and the current nonroot service user have read or directory read/search access; the owning group and others have none. The apparent group bits are the ACL mask. Additional principals, write access, default ACLs, malformed or unavailable ACL metadata, links and writable credential mounts are refused. File and directory must be on the same read-only filesystem, reached through root-owned ancestors with no nonroot write access or ACLs. CREDENTIALS_DIRECTORY must name the canonical absolute /run/credentials/<unit>.service directory and agree with any explicit --credentials-dir; an environment claim alone is insufficient. Manual owner-only credentials remain supported outside that context. The private state, baseline and lock guards receive no ACL exception.

The systemd credential contract describes service-user access and immutability. This reader accepts the restricted ACL/mount shape verified on systemd255, not every fallback or user-service layout. Before upgrading the observer, verify the exact accepted source and run a bounded transient service with a disposable, nonsecret LoadCredential input under the same service identity and sandbox. It must validate the actual credential metadata without networking, state changes or printing its value. Preserve the result before the operator tries a silent shadow run. A local filesystem fixture proves neither the installed systemd context nor shadow/runtime acceptance.

Set notification_verification in config to notification.json only when the actual evidence exists. Shadow's service template loads MQTT alone. No credential value appears in argv, exception logs, persisted state or public documentation. Configuration is JSON, never sourced as shell code.

Select a broker credential with only the required CONNECT authority when available; this probe does not publish or subscribe. Any credential change on the live broker belongs to the operator. The default username preserves the old check; it is not a recommendation to expand its permissions.

The notification evidence object requires schema: 1, mode: "verified-platform-equivalent", the configured owner_id, the reported probe_contract, a private proof_reference, and all of these verified booleans:

existing_recipients_preserved
down_delivery_verified
recovery_delivery_verified
outside_laboratorium
title_and_status_routing_verified
dedup_retry_verified

These are operator attestations to external evidence. The program cannot derive them from an ingest response. It binds their content into active state and rejects changed delivery/source/probe contracts. Token rotation at the same source URL preserves the binding; another source URL requires explicit reconciliation. Do not edit configuration or evidence during a tick; stop the observer unit first.

alert_ack requires the verified exact http_status, a proof_reference, and one body rule: empty, json-exact (with a body object), or json-required (with a nonempty body object containing the required fields and exact value types). Other status codes, redirects, transport failures, malformed/oversized bodies or a failed body rule leave the transition pending. The program does not infer a success rule from an unverified public example.

State and acknowledgement

Observations and notification acknowledgements are separate. A DOWN observation creates an outbox event and records the down-since time. Its acknowledged status remains resolved until the configured endpoint accepts the event and the local receipt is fsynced. A recovery never erases an unacknowledged DOWN; both transitions stay in order. A failed resolved request leaves the acknowledged outage open and retries later.

Each pending intent is atomically persisted and fsynced before any POST. A process restart reuses the same event ID and dedup key. A response receipt stores HTTP status and a response-body hash, never the raw provider body. Each accepted receipt is fsynced before the program claims acknowledgement. A lock serializes active runs and baseline import. One oldest event per probe is attempted per tick, in parallel across probes, so one failing channel cannot multiply a full job's latency by eight.

All four versioned inputs require the exact JSON integer schema: 1; true, 1.0 and "1" are refused. Stored state, its baseline envelope, observations, pending events and ACK receipts have checked field sets and types. Their timestamps must be valid UTC timestamps. activated_at is either the literal null initial marker or a UTC timestamp within the bound baseline's 15-minute activation window. Other falsy or malformed values are corrupt state, never a request to create or normalize an activation.

For an unactivated owner, baseline freshness is checked before probes, again after probes immediately before intent persistence, and after that persistence before delivery. Each initial POST worker checks it again immediately before HTTP dispatch, after reading its configuration and credentials. First activation is persisted only after a successful tick while the baseline is still fresh. Failed or expired initial ticks keep the null marker and all durable pending events or receipts; they cannot silently establish an owner exempt from the freshness check. If some POSTs were accepted before a later failure or expiry, retain those receipts and reconcile them upstream. Never delete state to restart the cutover. A legitimately activated owner can restart and continue after its initial baseline ages; the 15-minute requirement is for first activation.

This is at-least-once transport. If the receiver accepts an event but its response or the local acknowledgement write is lost, the same event is retried. Preventing another logical DOWN/UP then depends on the verified provider dedup behavior for the original key. The code does not claim exactly-once email delivery or that an asynchronous ingest acknowledgement is a completed status-page update.

The outbox holds at most 64 transitions per probe and the complete state file is bounded to 1 MiB. Reaching that limit, corrupt state, an expired initial baseline or changed binding fails closed for operator reconciliation. No transition is silently dropped and no state is automatically reset. Preserve state during upgrades and include it in the observer's own recovery scope.

Shadow and dry-run

The packaged example works for shadow once the MQTT credential exists; its deliberate activation gates stay unset. Review and install the shadow unit/timer templates only. Shadow observes the real public probes and writes its own shadow.json; it never initializes active state, reads the alert credential, posts events or sends mail. Its MQTT CONNECT is a real authenticated connection. Compare several completed samples with the station's contemporaneous observations.

For an entirely offline dry-run, provide an observations JSON object with exactly the eight keys. Each value has status: "firing" | "resolved" and a short safe detail label. Against an existing confirmed active state:

python3 -I /opt/oss-watchdog/watchdog.py \
  --config /etc/oss-watchdog/config.json \
  --state-dir /var/lib/oss-oo-watchdog \
  --input /path/to/observations.json dry-run

Dry-run performs no network call or credential-value read, needs no manual credential directory and writes no state. If CREDENTIALS_DIRECTORY is set, every mode checks that declared service context, including dry-run. Neither mode proves owner delivery or authorizes cutover.

One-owner cutover and rollback

The same eight keys must have one active owner. A local lock cannot stop a different machine's timer. The operator performs this bounded sequence:

1. Keep the station active while validating observer shadow, source acknowledgement semantics and independent recipient/delivery evidence. Confirm the exact package/config hashes, systemd version, secret file ownership and recovery custody. Once the actual delivery evidence exists, run validate without networking to record both contract hashes for the baseline. Do not install the active timer yet.
2. Stop the station timer and wait for its current service run to finish. Check for another station invocation and any in-flight ingest/mail. Keep its old script, units and marker files available for recovery.
3. Independently reconcile all eight upstream keys with the source request history and current incidents. Reconcile uncertain or failed sends and legacy markers; unknown status is a stop condition. Reconfirm owner notifications and the last acknowledged open/resolved state. Account for any route that auto-resolves or manually changes alerts.
4. Generate a fresh private baseline after the stop, with schema: 1, owner_id, probe_contract, activation_contract, UTC confirmed_at, proof_reference, and previous_owner_stopped, previous_run_finished, inflight_reconciled all true. Its probes object contains every exact key, each confirmed status, optional down_since, and per-key proof_reference. No entry may be inferred solely from a marker file or current HTTP health.
5. Using the same private credentials as the service, run validate to reconfirm the unchanged contract hashes without networking. Import the baseline using --input /path/to/baseline.json import-baseline. Import sends nothing and refuses to overwrite existing state. Complete initial activation within 15 minutes of baseline confirmation.
6. Run and inspect one active tick, then enable the active timer only after its real result passes. A continuing confirmed outage sends no new firing; its later recovery sends only resolved. All-green does not rewrite an unrelated or unknown incident baseline. Archive the cutover proof and verify the station remains stopped across reboot.

Use the service account and credential-directory context for manual validate/import operations; do not create root-owned active state that the unprivileged service cannot manage. Modes other than dry-run require --credentials-dir /path/to/private-credentials outside systemd.

To roll back ownership, stop the observer timer, wait for all workers to exit and reconcile its pending/acknowledged events upstream. Do not simply re-enable the old station timer with stale marker files: it can duplicate DOWN or lose a recovery. Restore a confirmed baseline for the selected owner before enabling exactly one timer. Preserve both histories. These steps remain necessary after a failed initial tick that might have posted some keys successfully.

Runtime limits and acceptance

The system templates run as oss-watchdog with read-only system/home access, private credentials and a 0700 state directory. Each probe and POST runs in a subprocess with a timeout, including DNS stalls; the parent kills and reaps a timed-out child. The job uses at most eight concurrent workers, a 256 MiB memory cap, no swap, 64 tasks, and a 90-second whole-job timeout (45 seconds for shadow). A calendar timer schedules each minute and retries after failures. An already-running oneshot does not overlap; a long tick can skip a scheduled minute. Exit 1 means pending delivery failures; exit 2 means configuration/local contract failure.

Validate templates against the observer's actual Ubuntu/systemd before enabling them. Verify timer activity, restart persistence, state ownership and runtime resource limits there. A provider-wide failure shared by the observer and monitored host, public DNS/TLS failure or an incident-platform outage remains outside a claim of full independence. The observer's own failure needs a separate liveness monitor; this package does not silently add one or claim to monitor itself externally.

Run local source tests with:

python3 -I watchdog/test_watchdog.py
python3 -I -O watchdog/test_watchdog.py
git diff --check

The tests use disposable loopback TLS/HTTP/WebSocket fixtures and no live provider, real monitored URL, production container or station timer. OpenSSL is used only to create a temporary test certificate. They cover actual CLI shadow/import/active transitions, exact eight-key preservation, HTTP errors/redirects and wrong bodies, lost acknowledgements, restart deduplication, queued recovery, missing credentials, corrupted/full state, atomic-write failures and one-owner locking. Refusal regressions cover all four schema boundaries and malformed stored activation/baseline envelopes, as well as baseline expiry during probes, intent persistence, POST-worker credential reads and completion after a known ACK. Valid fresh, exact 900-second and already-active restart controls stay supported. The MQTT fixture checks the actual CONNECT credentials, rejected authentication, invalid WebSocket and CONNACK responses, fragmentation and ping. Credential regressions check the captured ACL forms, foreign principals/write rights, malformed or unavailable metadata, protected ancestry and readonly mounts, real POSIX ACL masks, manual owner-only files and CLI context conflicts before probes. In a declared systemd context, direct HTTP probe workers also check the required MQTT file before outgoing work. Manual HTTP workers without that environment retain their independent HTTP-only behavior. The loopback fixture queues all eight concurrent connections before accepting TLS handshakes; this test capacity does not change any watchdog timeout. Local proof establishes source behavior; it does not establish live channel capability or complete the operator cutover gates. The native CI workflow installs Python/OpenSSL in its existing job container and runs this same suite in both modes. This does not enable an observer unit or replace the live shadow, delivery and ownership acceptance gates.