# Independent watchdog The watchdog package runs on a separate observer. It preserves the eight public probes and their incident deduplication keys. Shadow mode records observations without notification credentials. Active mode requires a confirmed cutover baseline and evidence that owner notifications survive a Laboratorium outage. This source change does not deploy a service or change the station timer. The example configuration deliberately blocks activation: its ingest acknowledgement contract and independent notification evidence are unset. Existing station marker files do not prove that an alert or email was accepted, because the old script changed them regardless of delivery. ## Preserved probe contract | Deduplication key | Exact alert title | Probe and expected result | |---|---|---| | `oss-oo-probe-landing` | oss-oo.io probe landing | `/about/`, HTTP 200 | | `oss-oo-probe-web-login` | oss-oo.io probe web login | `/user/login`, HTTP 200 | | `oss-oo-probe-web-healthz` | oss-oo.io probe healthz | `/api/healthz`, HTTP 200 | | `oss-oo-probe-api` | oss-oo.io probe api | `/api/v1/version`, HTTP 200 | | `oss-oo-probe-api-mcp` | oss-oo.io probe api mcp | `/mcp/healthz`, HTTP 200 | | `oss-oo-probe-oci` | oss-oo.io probe oci | `/v2/`, HTTP 401 | | `oss-oo-probe-git` | oss-oo.io probe git hosting | `/OSS-OO/laboratory.git/info/refs?service=git-upload-pack`, HTTP 200 | | `oss-oo-probe-events` | oss-oo.io probe event bus | `events.oss-oo.io`, verified TLS WebSocket `/mqtt`, authenticated MQTT 3.1.1 CONNACK accepted | HTTP probes do not follow redirects or use environment proxies. An OCI 401 is healthy only for its own probe; other checks still require 200. Missing MQTT credentials are a configuration failure, not a skipped eighth probe. MQTT has no fixed-IP DNS fallback. Each connection uses a different client ID, so shadow cannot disconnect the station's probe at the broker. The existing `metadata.source` identifier remains `station-watchdog` for route compatibility. `metadata.observer` identifies the new owner. The identifier is a compatibility field, not a claim that the observer runs on the station. Verify any provider filters before cutover. ## Independent delivery is a prerequisite The old ops-mail helper enters a container on Laboratorium and submits SMTP to its local relay, which forwards through Graph. The observer must not call this helper through SSH, use the station's administration key, or copy the broad mail application credential. The implemented channel is direct incident-source HTTPS ingest with separately verified platform notifications. It can replace the old parallel ops-mail only after the operator verifies equivalent DOWN and recovery delivery to the existing recipients, exact title/status routing, status-page behavior, and retry deduplication. A successful ingest response alone proves none of these outcomes. Until that evidence exists, leave the station active and use shadow only. If the existing platform cannot provide equivalent notifications, cutover stays blocked. An independently scoped mail delivery channel and separate per-channel receipts would be additional reviewed work. No independent SMTP endpoint or platform notification capability is assumed by this package. Do not remove the parallel ops-mail requirement just to pass the activation check. The operator can inspect the existing alert-source setup/request history, route, notification workflows and recipient configuration without sending a test event. Recover the actual success HTTP status/body rule and the current status of all eight keys there. If read-only evidence cannot establish recipient equivalence and actual delivery, the owner must choose and authorize either a bounded notification drill or a separately scoped independent mail channel. Do not manufacture the evidence file from the fact that credentials exist. The official [incident alert setup](https://docs.incident.io/alerts/getting-started) and [routing documentation](https://docs.incident.io/articles/4007103429-creating-escalations-and-incidents-from-alerts) describe ingest, escalation and incident creation as distinct configuration steps. The [official HTTP integration example](https://incident.io/blog/opsgenie-integrations-migration-guide) documents the bearer-authenticated JSON fields. These references do not establish the installed account's response schema, plan capabilities, recipients or delivery outcome. ## Files and privileges Install the two Python modules as root-owned, read-only files under `/opt/oss-watchdog/`. The service account needs no shell administration key, sudo, Docker socket or access to monitored-host files. | Generic path | Purpose | |---|---| | `/etc/oss-watchdog/config.json` | Nonsecret configuration, based on `watchdog/config.example.json` | | `/etc/oss-watchdog/mqtt.json` | Private `{"password": ""}` | | `/etc/oss-watchdog/alert.json` | Private `{"url": "", "token": ""}` | | `/etc/oss-watchdog/notification-verification.json` | Private operator evidence attestation | | `/var/lib/oss-oo-watchdog/active-state.json` | Durable active observations, pending transitions and acknowledgements | | `/var/lib/oss-oo-watchdog-shadow/shadow.json` | Shadow observations; never an acknowledged baseline | Secret and evidence inputs must be regular files, single-linked and owner-only readable (0600 or 0400 at the source). systemd `LoadCredential` makes them available as `mqtt.json`, `alert.json` and `notification.json` in its private credential directory. The credential-only reader supports the observed systemd255 root:root 0440 files and 0550 directory: exactly root and the current nonroot service user have read or directory read/search access; the owning group and others have none. The apparent group bits are the ACL mask. Additional principals, write access, default ACLs, malformed or unavailable ACL metadata, links and writable credential mounts are refused. File and directory must be on the same read-only filesystem, reached through root-owned ancestors with no nonroot write access or ACLs. `CREDENTIALS_DIRECTORY` must name the canonical absolute `/run/credentials/.service` directory and agree with any explicit `--credentials-dir`; an environment claim alone is insufficient. Manual owner-only credentials remain supported outside that context. The private state, baseline and lock guards receive no ACL exception. The [systemd credential contract](https://systemd.io/CREDENTIALS/) describes service-user access and immutability. This reader accepts the restricted ACL/mount shape verified on systemd255, not every fallback or user-service layout. Before upgrading the observer, verify the exact accepted source and run a bounded transient service with a disposable, nonsecret `LoadCredential` input under the same service identity and sandbox. It must validate the actual credential metadata without networking, state changes or printing its value. Preserve the result before the operator tries a silent shadow run. A local filesystem fixture proves neither the installed systemd context nor shadow/runtime acceptance. Set `notification_verification` in config to `notification.json` only when the actual evidence exists. Shadow's service template loads MQTT alone. No credential value appears in argv, exception logs, persisted state or public documentation. Configuration is JSON, never sourced as shell code. Select a broker credential with only the required CONNECT authority when available; this probe does not publish or subscribe. Any credential change on the live broker belongs to the operator. The default username preserves the old check; it is not a recommendation to expand its permissions. The notification evidence object requires `schema: 1`, `mode: "verified-platform-equivalent"`, the configured `owner_id`, the reported `probe_contract`, a private `proof_reference`, and all of these verified booleans: ```text existing_recipients_preserved down_delivery_verified recovery_delivery_verified outside_laboratorium title_and_status_routing_verified dedup_retry_verified ``` These are operator attestations to external evidence. The program cannot derive them from an ingest response. It binds their content into active state and rejects changed delivery/source/probe contracts. Token rotation at the same source URL preserves the binding; another source URL requires explicit reconciliation. Do not edit configuration or evidence during a tick; stop the observer unit first. `alert_ack` requires the verified exact `http_status`, a `proof_reference`, and one body rule: `empty`, `json-exact` (with a `body` object), or `json-required` (with a nonempty `body` object containing the required fields and exact value types). Other status codes, redirects, transport failures, malformed/oversized bodies or a failed body rule leave the transition pending. The program does not infer a success rule from an unverified public example. ## State and acknowledgement Observations and notification acknowledgements are separate. A DOWN observation creates an outbox event and records the down-since time. Its acknowledged status remains resolved until the configured endpoint accepts the event and the local receipt is fsynced. A recovery never erases an unacknowledged DOWN; both transitions stay in order. A failed resolved request leaves the acknowledged outage open and retries later. Each pending intent is atomically persisted and fsynced before any POST. A process restart reuses the same event ID and dedup key. A response receipt stores HTTP status and a response-body hash, never the raw provider body. Each accepted receipt is fsynced before the program claims acknowledgement. A lock serializes active runs and baseline import. One oldest event per probe is attempted per tick, in parallel across probes, so one failing channel cannot multiply a full job's latency by eight. All four versioned inputs require the exact JSON integer `schema: 1`; `true`, `1.0` and `"1"` are refused. Stored state, its baseline envelope, observations, pending events and ACK receipts have checked field sets and types. Their timestamps must be valid UTC timestamps. `activated_at` is either the literal null initial marker or a UTC timestamp within the bound baseline's 15-minute activation window. Other falsy or malformed values are corrupt state, never a request to create or normalize an activation. For an unactivated owner, baseline freshness is checked before probes, again after probes immediately before intent persistence, and after that persistence before delivery. Each initial POST worker checks it again immediately before HTTP dispatch, after reading its configuration and credentials. First activation is persisted only after a successful tick while the baseline is still fresh. Failed or expired initial ticks keep the null marker and all durable pending events or receipts; they cannot silently establish an owner exempt from the freshness check. If some POSTs were accepted before a later failure or expiry, retain those receipts and reconcile them upstream. Never delete state to restart the cutover. A legitimately activated owner can restart and continue after its initial baseline ages; the 15-minute requirement is for first activation. This is at-least-once transport. If the receiver accepts an event but its response or the local acknowledgement write is lost, the same event is retried. Preventing another logical DOWN/UP then depends on the verified provider dedup behavior for the original key. The code does not claim exactly-once email delivery or that an asynchronous ingest acknowledgement is a completed status-page update. The outbox holds at most 64 transitions per probe and the complete state file is bounded to 1 MiB. Reaching that limit, corrupt state, an expired initial baseline or changed binding fails closed for operator reconciliation. No transition is silently dropped and no state is automatically reset. Preserve state during upgrades and include it in the observer's own recovery scope. ## Shadow and dry-run The packaged example works for shadow once the MQTT credential exists; its deliberate activation gates stay unset. Review and install the shadow unit/timer templates only. Shadow observes the real public probes and writes its own `shadow.json`; it never initializes active state, reads the alert credential, posts events or sends mail. Its MQTT CONNECT is a real authenticated connection. Compare several completed samples with the station's contemporaneous observations. For an entirely offline dry-run, provide an observations JSON object with exactly the eight keys. Each value has `status: "firing" | "resolved"` and a short safe `detail` label. Against an existing confirmed active state: ```bash python3 -I /opt/oss-watchdog/watchdog.py \ --config /etc/oss-watchdog/config.json \ --state-dir /var/lib/oss-oo-watchdog \ --input /path/to/observations.json dry-run ``` Dry-run performs no network call or credential-value read, needs no manual credential directory and writes no state. If `CREDENTIALS_DIRECTORY` is set, every mode checks that declared service context, including dry-run. Neither mode proves owner delivery or authorizes cutover. ## One-owner cutover and rollback The same eight keys must have one active owner. A local lock cannot stop a different machine's timer. The operator performs this bounded sequence: 1. Keep the station active while validating observer shadow, source acknowledgement semantics and independent recipient/delivery evidence. Confirm the exact package/config hashes, systemd version, secret file ownership and recovery custody. Once the actual delivery evidence exists, run `validate` without networking to record both contract hashes for the baseline. Do not install the active timer yet. 2. Stop the station timer and wait for its current service run to finish. Check for another station invocation and any in-flight ingest/mail. Keep its old script, units and marker files available for recovery. 3. Independently reconcile all eight upstream keys with the source request history and current incidents. Reconcile uncertain or failed sends and legacy markers; unknown status is a stop condition. Reconfirm owner notifications and the last acknowledged open/resolved state. Account for any route that auto-resolves or manually changes alerts. 4. Generate a fresh private baseline after the stop, with `schema: 1`, `owner_id`, `probe_contract`, `activation_contract`, UTC `confirmed_at`, `proof_reference`, and `previous_owner_stopped`, `previous_run_finished`, `inflight_reconciled` all true. Its `probes` object contains every exact key, each confirmed `status`, optional `down_since`, and per-key `proof_reference`. No entry may be inferred solely from a marker file or current HTTP health. 5. Using the same private credentials as the service, run `validate` to reconfirm the unchanged contract hashes without networking. Import the baseline using `--input /path/to/baseline.json import-baseline`. Import sends nothing and refuses to overwrite existing state. Complete initial activation within 15 minutes of baseline confirmation. 6. Run and inspect one active tick, then enable the active timer only after its real result passes. A continuing confirmed outage sends no new firing; its later recovery sends only resolved. All-green does not rewrite an unrelated or unknown incident baseline. Archive the cutover proof and verify the station remains stopped across reboot. Use the service account and credential-directory context for manual `validate`/import operations; do not create root-owned active state that the unprivileged service cannot manage. Modes other than dry-run require `--credentials-dir /path/to/private-credentials` outside systemd. To roll back ownership, stop the observer timer, wait for all workers to exit and reconcile its pending/acknowledged events upstream. Do not simply re-enable the old station timer with stale marker files: it can duplicate DOWN or lose a recovery. Restore a confirmed baseline for the selected owner before enabling exactly one timer. Preserve both histories. These steps remain necessary after a failed initial tick that might have posted some keys successfully. ## Runtime limits and acceptance The system templates run as `oss-watchdog` with read-only system/home access, private credentials and a 0700 state directory. Each probe and POST runs in a subprocess with a timeout, including DNS stalls; the parent kills and reaps a timed-out child. The job uses at most eight concurrent workers, a 256 MiB memory cap, no swap, 64 tasks, and a 90-second whole-job timeout (45 seconds for shadow). A calendar timer schedules each minute and retries after failures. An already-running oneshot does not overlap; a long tick can skip a scheduled minute. Exit 1 means pending delivery failures; exit 2 means configuration/local contract failure. Validate templates against the observer's actual Ubuntu/systemd before enabling them. Verify timer activity, restart persistence, state ownership and runtime resource limits there. A provider-wide failure shared by the observer and monitored host, public DNS/TLS failure or an incident-platform outage remains outside a claim of full independence. The observer's own failure needs a separate liveness monitor; this package does not silently add one or claim to monitor itself externally. Run local source tests with: ```bash python3 -I watchdog/test_watchdog.py python3 -I -O watchdog/test_watchdog.py git diff --check ``` The tests use disposable loopback TLS/HTTP/WebSocket fixtures and no live provider, real monitored URL, production container or station timer. OpenSSL is used only to create a temporary test certificate. They cover actual CLI shadow/import/active transitions, exact eight-key preservation, HTTP errors/redirects and wrong bodies, lost acknowledgements, restart deduplication, queued recovery, missing credentials, corrupted/full state, atomic-write failures and one-owner locking. Refusal regressions cover all four schema boundaries and malformed stored activation/baseline envelopes, as well as baseline expiry during probes, intent persistence, POST-worker credential reads and completion after a known ACK. Valid fresh, exact 900-second and already-active restart controls stay supported. The MQTT fixture checks the actual CONNECT credentials, rejected authentication, invalid WebSocket and CONNACK responses, fragmentation and ping. Credential regressions check the captured ACL forms, foreign principals/write rights, malformed or unavailable metadata, protected ancestry and readonly mounts, real POSIX ACL masks, manual owner-only files and CLI context conflicts before probes. In a declared systemd context, direct HTTP probe workers also check the required MQTT file before outgoing work. Manual HTTP workers without that environment retain their independent HTTP-only behavior. The loopback fixture queues all eight concurrent connections before accepting TLS handshakes; this test capacity does not change any watchdog timeout. Local proof establishes source behavior; it does not establish live channel capability or complete the operator cutover gates. The native CI workflow installs Python/OpenSSL in its existing job container and runs this same suite in both modes. This does not enable an observer unit or replace the live shadow, delivery and ownership acceptance gates.