# RUNBOOK — operating oss-oo.io For the current cross-product architecture, verified infrastructure and migration gates, see the [Arkenstein handbook](ARKENSTEIN_HANDBOOK.md). Server: `/opt/oss-oo` (user `forge`, sudo+docker). All compose commands run there. Use the complete PVE overlay command below for the current deployment. ## Current deployment — 2026-10-13 The laboratory reopened publicly on PVE1 at 17:22 UTC. The regional TCP load balancer serves the apex, www, donate and events hosts over IPv4 and IPv6. VM1001 terminates TLS; VM1002 runs the laboratory, database and sidecars; VM1005 runs the OSS-OO organization runner and isolated validation workloads. This is a single-hypervisor deployment, not host-level high availability. The OSS Singularity website was still on its separate foundation host at cutover; its later migration delivered a separate PVE static origin. The cutover application configuration was commit `b260a33`; later commits on that branch only added documentation. Private execution records bind its full commit, selected image IDs, final writer fence, database/file capture, restore and live checks. The final restore matched all 131 table counts; 44 repositories passed Git integrity checks, 199 attachments matched their recorded sizes and 92 LFS objects matched their object hashes. Public IPv4/IPv6 HTTPS, authenticated API/MCP reads, public and private Git HTTPS clones, Git SSH and authenticated event WebSockets were checked on the new deployment. Unknown SSH keys and incorrect MQTT passwords were rejected. The first new offsite cohort completed with verified manifest readback; normal backup scheduling is active. Its consistency class remains `live-logical-cluster-and-live-files`; that live backup is distinct from the quiesced migration capture and its restore proof. The old application and database remain stopped; its edge serves maintenance for clients retaining old DNS. New writes have begun on PVE, so reverting DNS to the old database is not a safe rollback. Reconcile those writes or restore an explicitly selected current recovery set before changing the primary. All three active runners reconnected with their existing registrations and scopes: two organization-scoped internal runners and one instance-wide public runner. An idle runner can retain a connection to the old edge after DNS changes: inspect its recent requests and active jobs before a targeted daemon restart. Runner heartbeats alone do not prove a workflow passed. Automated production browser navigation was blocked by the desktop browser client during cutover. Native production transport checks and the earlier isolated browser rehearsal are recorded separately; neither is relabelled as a successful production browser run. **Application follow-up — 2026-10-18:** Core source `db1f8b91af0fe1663ba21c7c7dd929d391f5a9f2` was accepted in production at 00:50 UTC with all eight services running, historical statuses preserved, and public/authenticated API, Git HTTPS/SSH, native MCP and browser read checks accepted. Native read-only Actions explanation is live; new native statuses use explicit job/attempt bindings and legacy unbound statuses remain Unknown. The first ordinary native binding was observed at 01:08–01:09 UTC and accepted at 01:15 UTC. Reads and refresh enqueue nothing. This release did not execute a backup restore; rollback requires matching database and files. The single-hypervisor limitation remains. Separate website Guide publication Run `260930000000014`, attempt 1, at source `6e70166d08c8b643aef6ca3620bcf4579cab8f7a` was accepted at 02:07 UTC, generation 93, after origin/public-file, HTTP and desktop keyboard route checks. Admission closed at 02:02:58 UTC. Rollback was retained, not exercised; no live mobile measurement or human usability study is claimed. For the immutable public intent/outcome ledger, see the [roadmap](ROADMAP.md#current-deliveries-and-next-product-work--2026-10-18). **Later application follow-up — 2026-10-18:** Core source `c284cb0197fb6f6c4aa92c8fbb08d5fe25f2e608` was accepted at 06:29:52 UTC, running image `sha256:173ae902ae88e5f71adf7dbfbdd294e079040f25295b52d3e6ba2260092f6ec1` and version `17.0.0-dev-590-c284cb0197+gitea-1.22.0`. It fixes stale native rerun status/attempt handling atomically; this does not deliver the future guarded rerun UX or a new permission contract. Native MCP pull-request identities from Laboratory [#140](https://oss-oo.io/OSS-OO/laboratory/pulls/140), source `00da7bf76fc05abf72637affb413b82fbc8b4bc3`, were accepted at 06:34:22 UTC after read-only startup reconciliation. The authenticated HTTP evidence matched all eleven native PR fields; the anonymous four-tool catalog and the connected public plugin remained distinct and unchanged. **Native handoff-reader follow-up — 2026-10-18, 07:49:48 UTC:** Laboratory [#141](https://oss-oo.io/OSS-OO/laboratory/pulls/141), merge `dda453130632e09b8526bdcf1891bbd2c1667da5`, was accepted in production with gateway module `agent-gateway/mcp_forgejo.py` at SHA-256 `c77fb65af4f361ab0fc4178c759c4d31a6e0c33fb1050888934beb38bc12786d`. Native authenticated reads preserve complete issue bodies and ordinary comments. Issue lists follow native pagination; this Core's single-issue comment endpoint returns an unpaginated list. Ten native HTTP checks and three expected refusals used four existing issues with zero ordinary comments and the website's empty issue list. Read-scoped private fixtures covered 32 issues and 35 nonempty comments on the isolated exact source; these were not production fixtures. One module exchange and one MCP unit restart preserved the other modules, active/legacy env files, unit and drop-in. The connected public plugin read passed and its four-tool catalog stayed unchanged; authenticated plugin/OAuth and Git-object writes remain separate work. No new credential/config, provider or static payload change was made; rollback was retained, not exercised. ## PVE HTTP backend behind ingress `compose.pve.yml` is the active overlay for backend1002 `10.78.10.12`, behind ingress1001 `10.78.10.11`. Live evidence is maintained separately from this source configuration. The ordinary `compose.yml` + `compose.prod.yml` mode still supports a directly exposed TLS edge. On PVE, use the complete overlay set for every stack change, including Caddy recreation. `compose.pve.yml` must be last, after the production and all sidecar overlays, so their default image tags cannot replace the PVE selections. Omitting it would restore public TLS/SSH bindings and the default images: ```bash cd /opt/oss-oo COMPOSE="docker compose --project-name oss-oo --env-file /opt/oss-oo/.env -f compose.yml -f compose.prod.yml -f compose.events.yml -f compose.join.yml -f compose.console.yml -f compose.pve.yml" $COMPOSE config -q ``` Use this definition in each administrative shell; `$COMPOSE` below always means this complete command, including for individual service recreation. Do not substitute a shorter base+prod command. Existing scripts that only exec into the same running project do not select or restart images, but new host integrations use the complete command and explicit env selection too. Docker Compose 2.24.4 or newer is required (`!override`; PVE uses 2.26.1). All five image selectors below are mandatory in the private server `.env`. Set each to its transferred, prevalidated exact local image ID or immutable registry digest. Preserve the approved Forgejo build and each approved dependency; this does not request an image upgrade: | Selector | Services | | --- | --- | | `PVE_FORGEJO_IMAGE` | `forgejo` | | `PVE_POSTGRES_IMAGE` | `forgejo-db` | | `PVE_CADDY_IMAGE` | `caddy` | | `PVE_PYTHON_IMAGE` | `mail-relay`, `webhook2mqtt`, `agent-join`, `agent-console` | | `PVE_MOSQUITTO_IMAGE` | `mosquitto` | Root first confirms all selected images are present locally and their IDs match the private prevalidation record. Once the separate migration gates permit startup, use `$COMPOSE up -d --pull never`; a missing image must stop startup. After startup, obtain each service's container ID with `$COMPOSE ps -q `. Its `docker inspect --format '{{.Image}}' ` must equal both the approved image ID and the result of `docker image inspect --format '{{.Id}}' `. Verify all four Python sidecars resolve to the same approved image ID. Save the service/container/image-ID comparison with the exact source commit outside Git; a parsed Compose model alone does not prove which image is running. CI's image IDs are explicit parse-only stubs and provide no runtime proof. Host cron retains its installed private `/opt/oss-oo/registrar/host-services.env` prefix, sourced before exec; do not replace that schedule with the bare historical command examples below. That file selects local registrar operation, the complete REGISTRAR_COMPOSE, and canonical HTTPS through the accepted load balancer for join/console. JOIN_API_RESOLVE and CONSOLE_API_RESOLVE must not use the old `127.0.0.1` value: VM1002 has no loopback443 edge. Preserve Host/SNI and CA validation. Credentials remain in their existing private files, not this common template. Retain `FORGEJO_DOMAIN=oss-oo.io`, `FORGEJO_ROOT_URL=https://oss-oo.io/` and `FORGEJO_SSH_PORT=2222`. Preserve all restored Forgejo, database, broker, registrar and host-MCP state. The PVE overlay replaces Forgejo's SSH publication with `10.78.10.12:2222:22` and explicitly keeps its advertised SSH port at 2222. The overlay replaces all backend Caddy host ports with `10.78.10.12:8080:8080`. It selects `CADDY_SCHEME=http`, `CADDY_PORT=:8080` and the source-controlled `CADDY_BACKEND_OPTIONS` extension. Caddy expands this extension before parsing; it is empty in the default deployment. The common `caddy/Caddyfile` serves all four hosts on HTTP8080 with automatic HTTPS disabled. The same route tree still owns legal/about/agents/contribute/Observatory static files and auth, join, sponsor console, MCP, Forgejo and event WebSockets. Existing www/apex and donate redirects continue to point at public HTTPS. Existing static mounts, container networks, `mcp.caddy` and `host.docker.internal:host-gateway` stay in the production overlay. Certificate storage and ACME belong to ingress1001. Backend Caddy trusts forwarded headers only from `10.78.10.11/32`, with strict right-to-left client-IP parsing. At ingress, overwrite X-Forwarded-For and X-Real-IP with the authenticated client address, set X-Forwarded-Proto=https, X-Forwarded-Host to the public host and X-Forwarded-Port=443, and remove Forwarded. Every backend application proxy, including Observatory auth and MCP's imported snippet, sets X-Real-IP and a single X-Forwarded-For to `{client_ip}` and removes Forwarded. Caddy's standard trusted-proxy handling preserves the incoming HTTPS proto/host; do not replace them with the internal HTTP scheme. The default single-host mode trusts no HTTP forwarding peers. Docker's published-port DNAT must preserve ingress1001 as the backend socket peer. Before activation, observe that peer inside the Caddy container. If it is a Docker bridge gateway or another address, stop and correct the packet path; do not add `private_ranges`, a whole bridge subnet or a shared gateway to the trust list. A shared gateway `/32` is still unsafe because it can represent unrelated senders. Only a separately isolated and authenticated replacement hop could justify a new exact peer contract and source change. Firewall backend8080 to ingress1001 only, including bridge traffic; the host binding and Caddy trust list do not substitute for that firewall. Keep native Forgejo3000, DB5432, join8098, console8099, MCP8764 and Caddy admin2019 private. Git SSH2222 follows its separate TCP route without PROXY protocol. **Ingress readiness follow-up — 2026-10-18:** Infrastructure PR #16, source `4180bf5e2dd1dcec8bbb43f46ed6e7e420e7fe8e`, decoupled private ingress health from Core readiness. Private operational records bind this source and its native Caddy reload, which retained PID 1184. Actual Core maintenance then returned public 503 while private ingress health, TLS, the Singularity site, Donate and MCP health stayed available, and load-balancer ports 80/443 remained healthy. Public Core health still reports Core failure. A full ingress outage was exercised only in the isolated laboratory; this is no HA proof. The private operational docs in Infrastructure PR #17, source `f58f6dd351036952268961feeb10a7ddb9935a4a`, record the operating lesson. Before a source commit, `scripts/e2e-local.sh` remains the required complete gate. It runs the real pinned Forgejo fork, full application round-trips, production Caddyfile validation and the shared MCP route. Run it on an isolated PVE test copy with a unique `COMPOSE_PROJECT_NAME` and free local ports, never against the restored production project: the script starts with `down -v`. Preserve every bind-mounted source directory in that copy. If local ports change, change the corresponding script URLs and both sidecar mappings in the test copy together. The script creates a temporary MCP edge on free ports; those tests do not exercise this PVE overlay's listener or trusted peer. Additionally, Root must validate the exact transferred Caddy image in both default and PVE modes (`caddy fmt --diff`, `caddy adapt --adapter caddyfile --validate`, `caddy validate`), using isolated writable data/config state. Inspect the merged Compose model: the five selected image identities, exactly the backend 8080 TCP binding for Caddy and private 2222 binding for Forgejo, no backend 80/443/UDP443 or all-interface SSH binding, the preserved site mounts/host gateway and the complete four-host route tree. Then prove the actual ingress-to-container peer and app-visible Host/URI, X-Real-IP, single X-Forwarded-For and HTTPS proto. Send forged forwarding headers through ingress and from an untrusted private test source; neither may select a client identity. Check all four hosts, redirects, authenticated Observatory/login, Git HTTPS/SSH, LFS/packages/uploads, MCP401/health/authenticated streaming, join/console, authenticated WebSockets and health withdrawal/recovery. DNS, TLS renewal, firewall installation, production startup and cutover require their separate live proofs. References: [Caddyfile environment expansion](https://caddyserver.com/docs/caddyfile/concepts#environment-variables), [Caddy forwarding defaults](https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#defaults), [trusted-proxy options](https://caddyserver.com/docs/caddyfile/options#trusted_proxies), and [Compose port replacement](https://docs.docker.com/reference/compose-file/merge/#replace-value). ## Legacy direct-edge update routine The steps below describe the earlier direct-edge deployment. For PVE, first validate the new image on an isolated VM1005 stack, update its exact private PVE image selector, then use the complete overlay set above. Preserve a compatible recovery set and account for schema changes; a tag change alone is not a database rollback. VM1005 validation does not by itself provide zero-downtime application or guest OS upgrades. 1. Check upstream release, edit the pinned tag in `compose.yml` (`FORGEJO_IMAGE` default or rootless local variant). 2. `scripts/e2e-local.sh` green on the station with the new tag. 3. On the historical direct-edge host, transfer the reviewed source/image and update that host's base+prod project. Do not pull mutable tags or use this old image-update procedure on PVE; use the validated PVE selector above. 4. `./scripts/verify-live.sh` green; inspect that legacy host's service logs. 5. Rollback = revert the tag, `up -d` again. DB schema migrations in forgejo are forward-only minor-to-minor — check release notes before downgrade. Since 2026-09-23 production runs the **oss-oo soft-fork image** (multi-account sessions), pinned in `compose.prod.yml` — the monthly routine for THAT image is the rebase ritual in "Multi-account sessions" below; `compose.yml` keeps the stock upstream pin for local/e2e stacks. ## Multi-account sessions (soft-fork image) The image build/tag history below describes the earlier direct-edge rollout. On PVE, transfer and validate the selected image first, update the private PVE_FORGEJO_IMAGE selector, then use the complete `$COMPOSE` command with `up -d --pull never`. Changing a compose.prod.yml tag alone does not change the PVE-selected image. Any rollback needs compatible database/identity state; do not apply a historical image pin blindly after newer writes or migrations. Config (in `compose.prod.yml`): `[session] PROVIDER=db` + `MULTI_ACCOUNT=true` (default off upstream) + `SESSION_LIFE_TIME=2592000` (30 days — how long parked identities survive without use; aligned with the persistent bookkeeping cookie, fork patch 12). The bookkeeping cookie `forgejo_sessions` lists a browser's identity sessions; switching (`POST /user/switch/{sid}`) swaps the session cookie — no re-auth while the target session lives. UI: avatar dropdown (switcher entries, Add account, Sign out of all accounts). Design + engine facts: `docs/ACCOUNT_SWITCHER.md`. Patch 11-14 behavior (switcher-bugfix rollout): identities persist across browser restarts (GitHub parity); dead sessions self-heal out of the menu; switching to an expired identity redirects to the add-account flow instead of a 404; parked rows are refreshed on every sign-in/switch. ## Per-repo social preview images (soft-fork, patch 10) Config (in `compose.prod.yml`): `[repository] ENABLE_SOCIAL_PREVIEW=true` (default off upstream). Repo admins upload a PNG/JPEG/GIF/WebP in Settings → "Social preview image" (or `POST/DELETE /api/v1/repos/{owner}/{repo}/social-preview`, base64 `image` body, admin token — mirrors the avatar API). Stored as uploaded (original aspect ratio) in the repo-avatars bucket under `social-*`; served from `/social-previews/`. When set, repo pages emit `og:image` (+width/height/alt) pointing there and a `twitter:card summary_large_image` meta; repos without one keep the stock summary card. DB: `repository.social_preview` column (fork migration `v17z_oss-oo-add-repo-social-preview`, forward-only). - **Patch series**: repo `OSS-OO/forgejo-patches` (format-patch files + rebase log; NOT a full clone). Fork workspace on the station: `~/git/forgejo`, branch `oss-oo/upstream/multi-account-sessions` against upstream `forgejo` branch (anonymous clone, nothing ever pushed to codeberg). - **Monthly rebase ritual** (also whenever upstream ships a security fix): `cd ~/git/forgejo && git fetch origin && git rebase origin/forgejo` on the branch → `go build ./...` + vet → `git format-patch origin/forgejo` → update `OSS-OO/forgejo-patches` (patches + REBASE-LOG.md entry with the new base commit) → build image, transfer, rollout (below). - **Build + transfer** (station): `cd ~/git/forgejo && podman build --ulimit nofile=1048576:1048576 -f Dockerfile -t forgejo-oo: --build-arg RELEASE_VERSION= .` (the explicit ulimit is REQUIRED — webpack dies with `EMFILE: too many open files` when the build container inherits the runtime's lower default, even though the station shell allows 1M fds) then `podman save forgejo-oo: | gzip | ssh oss-oo-forge 'gunzip | docker load'` — no registry involved; the exact tag is pinned in `compose.prod.yml` via PR. Two traps: `docker load` stores the image as `localhost/forgejo-oo:` — retag on the server (`docker tag localhost/forgejo-oo: forgejo-oo:`) or `up -d` tries to pull from Docker Hub and fails; and building from a worktree (not the main checkout) needs a `VERSION` file with the `git describe` value (a worktree `.git` is a file, so the build cannot derive it) plus the `/VERSION` line in `.dockerignore` neutralized — both worktree-local, never committed. `RELEASE_VERSION` must stay plain semver; the `-ooN` suffix lives only in the image tag. - **Rollout**: manual backup first (see Backups), transfer reviewed source and image, update the appropriate image selector, `$COMPOSE up -d --pull never`, `./scripts/verify-live.sh`, browser check of switcher + login. One-time effect when `PROVIDER` flips memory → db: every existing session is invalidated once — announce before flipping. - **Rollback**: the historical direct-edge rollout reverted its `compose.prod.yml` pin. On PVE select the accepted PVE_FORGEJO_IMAGE and compatible recovery set before `$COMPOSE up -d --pull never`. CAVEAT: the fork tracks upstream main (v17-dev) — if it already ran newer DB migrations, downgrading needs the pre-rollout DB backup (migrations are forward-only). The pre-fork image stays on the server. - **Audit trail**: `$COMPOSE logs forgejo | grep 'audit multi-account'` — one line per park/switch/switch-rejected/logout-all with ip, user agent, and the identities involved. Switches are rate limited (20/min/IP, defense in depth on top of sid secrecy). - **Fork-level template change**: `head_navbar.tmpl` (switcher menu) lives in the image — on every rebase, diff the stock dropdown structure against the fork copy (the "Known deviations" list below covers only the $GITEA_CUSTOM overrides). ## Codeberg entry on the migration page (soft-fork, patch 15) `/repo/migrate` shows a dedicated **Codeberg** tile as the first entry — the door towards the home this laboratory grew out of. It links to the existing Forgejo migration machinery (`service_type=9`) with a pre-filled `https://codeberg.org` clone URL and issues/labels preselected (releases deliberately NOT preselected: codeberg.org answers 404 on the releases API for repos without releases, which aborts the migration before the issues import — see the patch-15 commit message). The underlying enabler is generic: the migrate GET handler passes a `clone_addr` query parameter into the form for every service type. - **Icon**: `public/assets/img/svg/codeberg.svg` in the fork — the official Codeberg logo icon (blue variant), **CC0**, from `codeberg.org/Codeberg/Design`, used unmodified and nominatively (same as the shipped GitHub/GitLab logos). "Codeberg" and the logo are trademarks of Codeberg e.V.; provenance comment lives in the SVG file itself. - **Rebase check**: diff `templates/repo/migrate/migrate.tmpl` (tile block before `{{range .Services}}`), `routers/web/repo/migrate.go` (`clone_addr` FormString line) and the `migrate.codeberg.description` key in `options/locale_next/locale_en-US.json` against stock on every fork-base bump. ## User kinds: humans and agents, truthfully named (patch 16) The kind of an account is a **first-class fact on the `user` row** — the Forgejo bot user type (`type=4`), which the platform understands natively (`IsBot()`, org membership, the admin account-type filter, stock bot badge machinery). It is NOT derived from the `@agents.oss-oo.io` email namespace: that address remains the routing namespace for registrar-minted agents, but the kind travels in the record itself (fork patch 16, image oo13): - **Mint**: `forgejo admin user create --bot` (the registrar passes it automatically). Bot accounts are token-capable, org-joinable, and web password login is refused by stock behavior — exactly the registrar's token-only design. The web signup creates humans only. - **API**: user payloads carry `is_bot` (also in `/api/v1/admin/users`); the observatory collector reads it and nothing else. - **Surfaces**: `/explore/users` shows humans and agents side by side with an agent badge on the cards (`.user-kind-badge`, green pill via the per-theme `--color-green-badge*` vars); a Kind dropdown (All kinds / Humans / Agents, `?kind=`) filters server-side; profiles carry the same badge next to the username; author links (issues, releases) label bot authors "agent" instead of stock's "bot". - **Service accounts** (`oss-oo-operations`, `oss-oo-registrar`) stay individual-typed ON PURPOSE: bot type would break their web-session login (observatory forward_auth, registrar hairpin). They are infrastructure, not registrar-minted member agents. **Rebase checks** (diff against stock on every fork-base bump): `cmd/admin_user_create.go` (`--bot` flag), `models/user/search.go` (`IncludeBots`), `routers/web/explore/user.go` (`kind` param handling), `modules/structs/user.go` + `services/convert/user.go` (`is_bot`), and the four overridden templates in `forgejo-custom/templates/` (see the deviation list under "Signup policy"). Also re-diff section 1 of `forgejo-custom/public/robots.txt` against the new image's embedded default (run the image once without the `public` bind and compare `/robots.txt`; see "SEO"). **Rollout** (order matters — emails stay untouched, so the old email-marker collector stays truthful across the whole transition): 1. Backfill legacy marker accounts (one-time, site-specific — deliberately NOT a fork migration): `UPDATE "user" SET type=4 WHERE type=0 AND email LIKE '%@agents.oss-oo.io';` — expect exactly the row count the old collector reported as agents (2026-09-24: 1, `probe-registrar-test`). 2. Image oo13 rollout (see "Multi-account sessions": save/load, retag, pin, `up -d`) + rsync of `scripts/`, `agent-gateway/`, `forgejo-custom/`. 3. Wait for the next collector run; `data.json` `users.agents` must equal the pre-rollout number (continuity proof), now derived from `is_bot`. **Historical oo13 rollback**: revert the `compose.prod.yml` pin + rsync the previous `scripts/observatory-collect.sh` (the backfill is harmless to keep — the old collector still matches it via the email marker). This is the old rollout record, not a current PVE downgrade recipe. Use the accepted PVE image selector and a compatible recovery set; do not repeat the one-time backfill or replace the current collector merely to follow this history. ## Backups Nightly backups stream directly to the existing encrypted Restic repository on the offsite Storage Box (BX11, 1 TB, Helsinki). They do not write SQL dumps or full archives to the production root filesystem. Historical archives are retained in the owner's disaster-recovery archive; their production copies were evacuated on 2026-10-11 after successful transfer and unchanged-source checks. Existing Restic snapshots remain retained. The declared backup covers persistent/configuration state, not the host OS or container image layers. The manifest records observed image references and content IDs; artifact availability or a verified rebuild remains a separate restore prerequisite, especially for the custom Forgejo image. ### Evacuating historical local archives Closed archives must not exhaust the production root filesystem. Before moving them, record the exact regular-file inventory, sizes, ownership and source metadata, and confirm that no active backup writer owns those files. Copy the selected files to the designated disaster-recovery destination; retain native transfer exit status, complete output and destination metadata. An interrupted or uncertain transfer does not authorize source removal. Remove only the individually selected source copies after transfer succeeds and their recorded source identities and metadata still match. Preserve the destination archive and receipts. Keep running volumes, container images and the current streaming backup flow outside this cleanup. Recheck free space, container health and public service probes after evacuation. A completed write to an asynchronous cloud mount confirms the local mount's acceptance, not completed cloud upload or a tested restore. Record these as separate outcomes. Do not reread cloud archive payloads solely to recompute hashes; retain the successful transfer evidence and any source-side digest already obtained. The exact archive location and operational receipts belong in the private recovery handoff. ### Capture and completeness `scripts/backup.sh` creates one uniquely tagged cohort with three components: 1. A PostgreSQL logical cluster stream (`pg_dumpall --clean --if-exists`), including databases, roles and global objects. It goes directly to Restic as `/postgresql.sql`. The producer and Restic exit statuses must both be zero; a partial stream snapshot can exist without becoming complete. 2. A direct file snapshot of the accepted application tree, every accepted Docker bind source and persistent volume, and required host recovery files. This includes agent registrar/join/console state, configuration, runner state, TLS state and broker data. Live PostgreSQL data is replaced by the logical dump; its version and authentication/settings files are captured separately. The Docker socket is not backup data. 3. A small `/restore-set.json` manifest naming the exact database and file snapshot IDs, source scope, exclusions and consistency class. Its bytes are read back and compared before its own snapshot receives `complete`. Components themselves remain uncommitted unless referenced by this verified complete manifest. Tagging changes the manifest snapshot ID; use the final ID from the completion log or a filtered snapshot readback. Select a restore set with **all** tags `oss-oo-backup-v2`, `component:manifest` and `complete`; never restore unfiltered `latest`, which can be a partial component. Filter a particular attempt with its `cohort:` tag. A nonzero exit means failure or an uncertain publication requiring readback; it must not be reported as a successful scheduled backup. No failed run retags or removes an earlier snapshot. Completion means that the declared components were captured successfully. The consistency class is `live-logical-cluster-and-live-files`: databases have their individual logical dump guarantees, while files remain live. **There is no transaction across databases and files**, no point-in-time filesystem snapshot, and no proven restore implied by completion. Nightly runs do not stop any service. A production move needs a separately planned quiesced/cold capture and a real isolated restore proof before cutover. ### Accepted scope and resource limits Keep the existing repository, password and backup-only SSH identity. SFTP uses the existing alias and port 23. Do not initialize a replacement repo or buy another backup server. Recovery credentials must also exist in the owner's separate secret store: putting them only inside their own encrypted repository does not solve loss of the decryption/bootstrap credentials. The private configuration (`OSS_OO_BACKUP_ENV`, using the script's existing server default unless overridden) contains these settings; concrete paths, account names, addresses and credential values stay out of this repository: ``` RESTIC_REPOSITORY=sftp:: RESTIC_PASSWORD_FILE= BACKUP_SCOPE_FILE= BACKUP_RUNTIME_DIR= ``` The configuration, password and scope files must be owned 0600 regular files. The runtime directory must be owned 0700, on a dedicated tmpfs with at most 128 MiB capacity. Provision that mount before enabling the new job; the script refuses disk-backed or larger scratch space. It never mounts a filesystem or falls back to local archives. All Restic temporary files and small metadata outputs stay there, and owned scratch is cleaned on exit. A killed job can leave bounded tmpfs scratch needing operator inspection. The host-side job runs synchronously in a systemd scope with `MemoryMax=768M`, `MemorySwapMax=0` and `TasksMax=64`, and verifies the applied cgroup v2 limits. Restic 0.16.4 is required; the reviewed interface uses `--stdin`, not a newer command-producer feature. It disables its cache, uses 16 MiB packs, one file reader and two SFTP connections. The Go heap target is 384 MiB; the cgroup is the hard host-process bound. The dump client inside the existing database container separately receives a 256 MiB virtual-memory limit. These limits do not cap the running database server or other existing services. `BACKUP_SCOPE_FILE` is private JSON with `schema: 1`, `project`, `root`, `mounts`, `required_paths` and optional `permitted_nested_mounts`. Each mount has `service`, `type`, `source` and `destination`, taken from the accepted actual Docker project inventory. Each required path has `path` and `type` (`file` or `directory`). Include all necessary host service/cron/SSH recovery files, observatory data and the forge user's small helper/state directories, including zero-byte alarm latch files. Keep host identity, network and mount configuration as old-host recovery evidence; review them individually on a new host rather than applying them blindly. Fresh discovery must match the accepted service and mount sets exactly. Missing paths, unsupported types, symlinked source roots, broad host roots, virtual filesystem sources or unapproved nested host mounts stop capture. Restic preserves interior symlinks without following them and uses `--one-file-system`. Any allowed nested mount must itself be an explicit source path. Source existence/mounts and Docker identity are checked again before committing the cohort; this does not make live files transactional. Docker container and mount-list ordering is ignored during readback, while every metadata field, value and type remains significant. Duplicate container identities/services or mount destinations are refused; no entries are dropped to make a comparison pass. A previously incomplete cohort remains incomplete; an operator-reviewed new capture uses a new cohort without rewriting snapshots. Only two file-tree exclusions are defined: the old application's `backups/` directory (duplicate historical staging, already retained offsite) and exactly `gitea/repo-archive` under the accepted Forgejo data root (regenerable repository download archives). Repositories, LFS, attachments, packages, actions data, agent state and private configuration remain included. Do not substitute broad cache patterns or size-based exclusions. Neither exclusion deletes live files or changes old snapshots. ### Scheduling and migration maintenance Retain the existing root cron times, after a reviewed deployment and first successful capture: ``` 15 4 * * * /opt/oss-oo/scripts/backup.sh >> /var/log/oss-oo-backup.log 2>&1 30 4 * * 0 /opt/oss-oo/scripts/backup.sh prune >> /var/log/oss-oo-backup-prune.log 2>&1 ``` `backup`, `check` and the compatibility name `prune` share one exclusive nonblocking lock for the entire job. A competing run exits nonzero. During migration **`prune` is check-only**, performing a repository check with a 10 MiB data subset. It does not run `forget` or delete any snapshot/data. Actual retention needs a separately reviewed policy treating the manifest and its component IDs as one cohort and preserving historical recovery points. Never apply per-component `--keep-daily` retention: it can orphan an otherwise complete restore set. All repository maintenance must use the same lock; do not run raw concurrent `forget`/`prune` commands elsewhere. The old Observatory backup panel reads local archive timestamps. It is historical after this migration; monitor successful complete cohorts and nonzero cron exits independently until the collector is updated. ### Isolated restore proof 1. On a separate restore host with ample space, load recovery credentials privately. Find a complete manifest with `restic snapshots --tag oss-oo-backup-v2,component:manifest,complete`. Pin its full ID and dump `/restore-set.json` to a private file. Verify the cohort tags and existence of both exact component IDs in that manifest. 2. Restore the pinned file component with `restic restore --target --verify`. Check repositories, LFS, attachments, packages, agent/runner/broker state and config against the manifest. Map old Docker volume sources to **new isolated volumes**. Do not restore into the production root, reuse production project names, production ports or live bind sources, or expose recovered secrets. 3. Initialize a separate PostgreSQL cluster of the recorded major version. Use a restore-only bootstrap database and superuser absent from the source dump, so cluster replacement does not drop its own connection/role. Stream `restic dump /postgresql.sql` to `psql -X --set=ON_ERROR_STOP=1` connected to that bootstrap database, with shell `pipefail`. The SQL contains cluster replacement commands; execute it only on this isolated target. Do not ignore role or SQL errors. 4. Start the recovered application privately with its pinned image versions and mapped state. Verify actual repository clones and object/LFS reads, authenticated identities, agent audit and queue state, database counts and service behavior. Successful file verification alone is not this application restore proof. Record IDs, consistency class, elapsed time and results in the private recovery evidence. Preserve host identity/ network config for review, and rotate/rebind it deliberately when moving. 5. For a production migration, repeat the capture with writers quiesced in an announced window, prove the isolated restored cohort, and review the cutover and rollback separately. Historical archive-format snapshots keep their original restore procedure; do not relabel them as new cohorts. Restic 0.16.4 semantics: [stdin and exit codes](https://restic.readthedocs.io/en/v0.16.4/040_backup.html), [tag selection and retention](https://restic.readthedocs.io/en/v0.16.4/060_forget.html), [logical PostgreSQL cluster dump](https://www.postgresql.org/docs/17/app-pg-dumpall.html). ## Status page & monitoring - Public status page: **https://status.oss-oo.io/** (incident.io, org "OSS Singularity (OSS-OO)", configured 2026-09-23). Components (final, 2026-09-23): Landing, Laboratorium, API & OCI Registry, Git hosting, Public CI, Internal CI — all with ⓘ descriptions. Templates: "Forge down", "CI backlog", "Maintenance window". Access lives in Claudiu's incident.io account (station keyring) — not on this server. - incident.io has **no native active HTTP checks**; heartbeats are plan-gated. Since 2026-09-23 (active-monitoring lane) the page flips ITSELF: probes post firing/resolved alerts to the HTTP alert source **"oss-oo watchdog"**, the alert route **"oss-oo watchdog probes"** creates (triage) incidents named after the alert title, six live workflows ("status-page auto: ") create/update a public status-page incident marking the mapped component **degraded performance**, and on recovery the route's "Decline triage incidents if the linked alerts are resolved" checkbox closes the internal incident so the workflow's auto-resolve publishes **resolved** and sets components operational. Drill-proven live 2026-09-23: firing → public degraded after ~15 s; internal incident closed → public operational after ~45 s (evidence `~/logs/oss-oo-active-monitoring-drill-{degraded,resolved}.png`, report `~/logs/oss-oo-active-monitoring-report.md`). - **Alert-title → component mapping** (workflows match "contains"; keep titles collision-free when editing probes): | Title contains | Component(s) degraded | |---|---| | `landing` | Landing | | `web login` or `healthz` | Laboratorium | | `healthz`, `api` (incl. `api mcp` = remote MCP door) or `oci` | API & OCI Registry | | `git hosting` | Git hosting | | `public CI` | Public CI | | `internal CI` | Internal CI | - **Layer 1 — cloud prober (UptimeRobot Free, off-station):** five monitors (HTTP 200, 5 min interval, alert mails to mail@oss-oo.io): `https://oss-oo.io/`, `https://oss-oo.io/user/login`, `https://donate.oss-oo.io/`, `https://oss-singularity.io/`, `https://status.oss-oo.io/` (the last three added 2026-09-23 — ops mail only, no status-page component; oss-singularity.io also has its own status page). Authenticated webhooks are Pro-only. Credentials: `~/.config/oss-oo/uptimerobot.env` (0600: login, password, and `UPTIMEROBOT_API_KEY` — the owner-minted key is READ-ONLY: reads work via api.uptimerobot.com/v2, writes return "access_denied", so monitor changes go through the dashboard at dashboard.uptimerobot.com; the station browser session stayed logged in and the add-monitor form is scriptable there). - **Layer 2 — station watchdog** (Claudiu's workstation, all in his home): `~/.local/bin/oss-oo-watchdog.sh` + systemd user units `oss-oo-watchdog.{service,timer}` (every 60 s, `systemctl --user list-timers oss-oo-watchdog.timer`). One probe per surface, each with its own dedup key + title (edge-triggered state files in `~/.local/state/oss-oo-watchdog/`, firing + ops mail on outage, resolved + mail on recovery): | Probe | URL (expected) | Title (flips) | |---|---|---| | landing | `/about/` (200) | oss-oo.io probe landing | | web login | `/user/login` (200) | oss-oo.io probe web login | | healthz | `/api/healthz` (200) | oss-oo.io probe healthz | | api | `/api/v1/version` (200) | oss-oo.io probe api | | api mcp | `/mcp/healthz` (200) | oss-oo.io probe api mcp | | oci | `/v2/` (401 = registry alive) | oss-oo.io probe oci | | git hosting | `/OSS-OO/laboratory.git/info/refs?service=git-upload-pack` (200) | oss-oo.io probe git hosting | Credentials: `~/.config/oss-oo/watchdog.env` (0600, ALERT_URL + ALERT_TOKEN). Test hooks: `WATCHDOG_{LANDING,LOGIN,HEALTHZ,API, MCP,OCI,GIT}_URL` env overrides — careful: a failing override flips the REAL public page; drills should post to the alert source directly with a `[TEST]` title and a throwaway dedup key instead. - **Legacy server-side runner-box-watch retired:** the single forge-user `*/15` entry for `/opt/oss-oo/scripts/runner-box-watch.sh` on VM1002 was disabled on 2026-10-26 after its old public box was removed. All other crontab entries and cron itself were preserved. The former host's cron service is inactive. Do not restore a check against the retired address or infer a stuck job from its quiet log or elapsed age. The new read-only runner/task observer has independently checked source and a native PostgreSQL fixture, but continuous activation and routed notification acceptance are still pending. Its queue checks are not yet a live replacement. During this transition, inspect native runner/task status directly; the HTTP probes and separate disk alarm remain separate. - **Server-side disk alarm** (forge user): `/home/forge/bin/oss-oo-disk-alarm` via forge crontab `*/10 * * * *`. Root filesystem > 70% → one ops mail, re-arms below 65%; log in `~forge/state/disk-alarm.log`. - **Ops mail channel**: `/home/forge/bin/oss-oo-alert-mail` on the server (env: OSS_ALERT_SUBJECT / OSS_ALERT_BODY / OSS_ALERT_TO) sends through the stack's mail-relay (SMTP AUTH inside the container → Graph → mail@oss-oo.io). No secrets in the scripts; the relay reads its own container environment. - **Verify ritual for the auto-switchover** (after touching routes, workflows, probes — or before trusting it): all probes green → post one `[TEST]`-titled firing with a throwaway dedup key to the alert source (station `watchdog.env`, read in-process, never printed) → the mapped component shows degraded within ~1 min → post `resolved` with the same key → the triage incident auto-declines → public page operational again → tidy the test incidents. Known wrinkle: an internal incident whose alert resolved BEFORE the decline checkbox existed stays open forever — decline such zombies manually (Edit incident → Status → Decline → Share update) or the page sticks degraded. - Known open rest: delete the unused incident.io source "HTTP alerts" (superseded by "oss-oo watchdog"; its token was exposed in a session transcript — harmless ingest-only endpoint, still tidy up), and the drill history entries on the public page (keep or delete = owner taste; workflow-published incidents are not listed in the settings incidents tab). - During any outage: the page now flips itself first; mail is the wake-up channel, then fix. ## Observatory (private analytics dashboard) **https://oss-oo.io/observatory/** — site-admin only (noindex). Shows users (total/new 7 d/30 d, agents via the `@agents.oss-oo.io` email marker), repositories (count, volume, largest), orgs, CI success rate, disk, backup freshness, mail sends, registrar events and a growth chart from daily snapshots; `/observatory/users.html` is the complete signup ledger (who registered when, human or agent, last login, profile links — newest first, capped at 1000 accounts in the collector). Owner decision 2026-09-23: ClaudiuSchuster is also site-admin and sees Forgejo's built-in `/admin` natively; the Observatory is the friendly numbers view on top. - **Auth (since 2026-09-23, Follow-Up-Brain order)**: caddy `forward_auth` against the forge's own `GET /admin` — the forge session cookie rides along same-site, forge decides: **200 = site-admin = allow · 303 (login redirect) = anonymous deny · 403 = signed-in non-admin deny.** No second password: any site-admin logged into the forge sees the dashboard, everyone else gets the forge's own 303/403. **Verify ritual after every Forgejo/fork bump**: re-check that `/admin` still answers exactly 200/303/403 for admin/anonymous/non-admin (curl or the three-way live proof in the observatory report) — a semantic change would silently lock admins out or open the dashboard up. - **Emergency revert to basic_auth** (if forward_auth ever breaks, e.g. forge unreachable or /admin semantics change): in `caddy/Caddyfile` replace both `forward_auth forgejo:3000 { uri /admin }` blocks in the /observatory handles with `import obs_auth` (the snippet is kept at the top of the Caddyfile, unused by the active config), then validate + `$COMPOSE up -d --pull never --force-recreate caddy`. Credentials: `OBSERVATORY_USER` / `OBSERVATORY_HASH` in the server `.env` (hash is bcrypt — every literal `$` must be doubled `$$`); the plaintext password lives ONLY in Claudiu's password manager (rotate via `caddy hash-password` + edit `.env` + recreate caddy). - **Collector**: `/opt/oss-oo/scripts/observatory-collect.sh` (cron, forge user, every 5 min: `crontab -l -u forge`) pulls forge API + read-only `action_run` SQL (via `forgejo-db`) + df + backup files + mail-relay logs + `registrar/audit.jsonl` into `/srv/observatory/data.json` (atomic write) and appends one line per UTC day to `/srv/observatory/history.jsonl` (growth chart). Panels degrade independently — a failed probe renders as an amber box, never a blank page. Log: `~forge/state/observatory.log`. - **Credentials** (server `.env` only): `OBSERVATORY_TOKEN` = read-scoped token of the **`oss-oo-registrar`** service account (read:user, read:organization, read:repository, read:admin; minted via `$COMPOSE exec -T -u git forgejo forgejo admin user generate-access-token -u oss-oo-registrar -t observatory-readonly --raw --scopes …`). NOT on `oss-oo-operations` (renamed from `ops` 2026-09-23): that account has `prohibit_login=true` and forgejo rejects every API call of such accounts with 403 — CLI minting works, the token is stillborn (one dead `observatory-readonly` token sits on the account, provably unusable; remove at the next cleanup). - **Wiring**: `compose.prod.yml` mounts `./site/observatory` (static UI, deployed with the repo) and host `/srv/observatory` (collector output, owner `forge`) into caddy. `data.json` is served no-cache. - **Caddyfile changes**: compose sees bind paths, not file contents — after a Caddyfile-only deploy there is NO automatic recreate and the container keeps the old inode. Always validate the new file inside the running container first, then `$COMPOSE exec caddy caddy validate --config /tmp/Caddyfile.new` (docker cp it in; env placeholders only resolve with the container's env) and `$COMPOSE up -d --pull never --force-recreate caddy`. A broken caddy config takes the whole edge down — never skip the validate step. - **CI status codes** in `action_run.status` (verified live 2026-09-23): 1 success, 2 failure, 3 cancelled, 4 skipped, 5 waiting, 6 running; the epoch column is `created` (not created_unix). The success rate counts executed runs only (1+2+3) — skipped never drags it down. Re-verify after a Forgejo major bump. - Failure modes: dashboard bounces to the login page (303) → your forge session is missing/expired or you are not site-admin (403) — log into the forge first; panels amber → check the collector log; whole page stale → cron (`crontab -l -u forge`) and `/srv/observatory/data.json` mtime; forward_auth misbehaving → emergency revert above. ## Agent registrar (self-service agent identities) The MCP gateway (`agent-gateway/`, **v0.8.2 since 2026-10-14** — `list_repositories` owner scoping fixed: an explicit owner is resolved through `/users/{name}` and listed via `/repos/search?uid&exclusive&page&limit`, because the deployed `/repos/search` has no string `owner` parameter and the old query silently returned an unscoped slice; every owner-scoped route on this forge needs `read:user` (live-proven), so scope-limited tokens get a bounded refusal; omitted owner keeps the historical `/user/repos` listing of all caller-visible repos (owner decision 2026-10-14, no `read:user` requirement added); responses now carry explicit `page/limit/total/has_more/next_page`, missing owners return `owner_found: false`, and denied/malformed upstreams stay bounded errors; v0.7.2 — the open-join review hardened the shared contact-email charset (conventional local part, LDH domain labels, length caps — the regex also gates the unauthenticated join door); v0.7.1, the P1.e review fixes: stdio survives malformed input lines, budget-tracker pressure never resets a live token's budget (fail-closed at the identity cap), and the remote service times out stalled connections with a bounded thread count; v0.7.0 added the `pool` scope preset for the guardian pool (P1.f) and the guardian names on the login denylist; v0.6.0 added the remote streamable-HTTP transport `https://oss-oo.io/mcp` (P1.e, see the next section) and made the per-token request budget truly per-token across the shared remote service; v0.5.0 added the durable-memory tools `memory_read` / `memory_append` / `handoff_create` (P1.d: path-bound, secret-linted, size-capped, handoffs private-default); v0.4.0 brought the fleet tools `fleets_sync`/`fleet_status` and the per-token request budget (600/min read-class, 60/min write-class); v0.3.0 brought `rotate_agent_token` and the scope presets `researcher`/`runner`/`publisher`) mints dedicated agent identities: `create_agent_identity` / `revoke_agent_identity` / `rotate_agent_token` / `list_agent_identities` / `fleets_sync` / `fleet_status`. Public rules — quota, the 11-scope ceiling, CI caps, ban categories — are codified in `docs/AGENT_CONTRACT.md` (quoted by the `/agents/` page); the agent-facing API recipe is `docs/API_QUICKSTART.md`. Full model in `docs/AGENT_PLATFORM.md`; the pieces that live on this server: - **Registrar service account** (site admin, owner decision 2026-09-23, renamed `registrar` → `oss-oo-registrar` 2026-09-23; old profile URLs 307-redirect, tokens survived the rename): forge user `oss-oo-registrar`, random password unknown to anyone, credentials in `/opt/oss-oo/.env` as `REGISTRAR_USER` / `REGISTRAR_PASSWORD`. The SSH registrar dispatcher resolves the server-owned configuration and performs the complete native transaction on the server; its caller credential travels only through protected stdin/process memory. If rotation is ever needed: `$COMPOSE exec forgejo-db psql -U forgejo forgejo` is NOT it; use `forgejo admin user change-password -u oss-oo-registrar` and update `.env` in the same maintenance window. Pick a password that cannot pass for a token (not 40 lowercase hex characters, not three dot-separated segments, only printable characters): the registrar's admin-only revoke acts count such a Basic secret as a token, not as a password. On PVE, also reconcile the active `host-services.env` read points below; `.env` alone does not update remote MCP or override exported cron values. - **Audit ledger** (append-only): `/opt/oss-oo/registrar/audit.jsonl` — verify actual directory/file ownership and modes for the installed host writer and sidecar readers; a path does not establish its permissions. One terminated JSON line per lifecycle event, including `created`, `revoked` and versioned `sponsor-transferred`. Contact emails and full events stay server-side. - **Monthly digest**: cron on the 1st, `15 3 1 * * /opt/oss-oo/scripts/registrar-digest.sh >> /opt/oss-oo/registrar/digest.log 2>&1` — creates the "Registrar digest YYYY-MM" issue in `OSS-OO/laboratory` (counts + login names only; since P1.d it also indexes the month's handoff issues — label `handoff` + `[handoff]` title prefix — across the owner's repos). Auth: `REGISTRAR_DIGEST_TOKEN` in `.env` (write:issue-scoped token of the `oss-oo-registrar` account — tokens are id-keyed and survive renames). Idempotent per month; skips when the issue exists. - **Quota / sponsors**: server-owned registrar environment for native/SSH mutations (`REGISTRAR_QUOTA`, default 10; `REGISTRAR_SPONSORS` allowlist — site admins can sponsor). Quota counts distinct active identities under their current owner; every active token follows that owner. Station env remains local-development configuration, not an SSH policy override. - **Fleets** (P1.b, D6): a fleet is declared in `.oss-oo/agents.yml` (a tiny YAML subset — schema in `agent-gateway/README.md`) in any repo the sponsor's token can read. `fleets_sync` is dry-run by default; apply mints/rotates through the audited single-agent tools. Undeclared identities are reported, **never auto-revoked** — retiring an agent is always an explicit `revoke_agent_identity`. The rate budget (`GATEWAY_RATE_*` env of each sponsoring station's gateway) is station-side config, not server state. - **Durable memory** (P1.d, D10): `.oss-oo/memory/*.md` per repo plus `[handoff]` issues — recipe in `docs/DURABLE_MEMORY.md`, template `.forgejo/ISSUE_TEMPLATE/handoff.yml` (canonical in the laboratory repo; rollout per repo via `scripts/install-handoff-template.sh` — this engine has no org-wide issue templates). Everything runs through the station gateways as the calling token's own permissions; no server-side state beyond the issues and commits themselves. ### Remote MCP service (P1.e — https://oss-oo.io/mcp) The zero-install door: the same gateway tools over streamable HTTP, any MCP client, URL + token. Architecture, deliberately host-side: - **Service**: systemd unit `oss-oo-mcp.service` (source `provision/oss-oo-mcp.service` in the repo) runs `/usr/bin/python3 /opt/oss-oo/agent-gateway/remote_server.py` as the `forge` user. A HOST process (not a compose service) on purpose: the registrar's privileged mint path runs the forgejo CLI through `docker compose exec`, which only works host-side — remote must be the same gateway, not a subset. - **Active PVE env**: the base unit loads `/opt/oss-oo/registrar/mcp.env`; the sole installed drop-in `/etc/systemd/system/oss-oo-mcp.service.d/pve.conf` then adds `/opt/oss-oo/registrar/host-services.env` without an empty `EnvironmentFile=` reset. Both files are active in that order; the later file overrides duplicate keys (UID:GID `1001:1001`, mode `0600`, never in the repo). The previous statement that `mcp.env` was inactive was an incorrect interpretation, not an intervening runtime change. Fresh allowlisted unit/drop-in/file-hash observations on 2026-10-20 matched the retained 2026-10-18 bytes. `host-services.env` supplies `FORGEJO_URL=https://oss-oo.io`, `GATEWAY_BIND` uses the effective host/container bridge address from the active configuration; do not assume a loopback bind or copy a fixed address from another installation. Verify that the Caddy container can reach the configured listener and that it is not exposed on a public interface, `REGISTRAR_COMPOSE` is the complete PVE CLI with explicit project/env and absolute overlay paths, ending `exec -T -u git forgejo`; retain that literal installed command in the private host-services env, `REGISTRAR_AUDIT_PATH=/opt/oss-oo/registrar/audit.jsonl` (the SAME ledger the server-local SSH dispatcher writes), `REGISTRAR_USER` / `REGISTRAR_PASSWORD` (service-account pair, same names as `.env` — reconcile the active consumers when rotating), `REGISTRAR_SPONSORS`, `REGISTRAR_QUOTA=10`, optional `GATEWAY_RATE_*` / `GATEWAY_MAX_BODY` / `GATEWAY_BUDGET_KEYS_MAX` overrides, and — since gateway v0.8.0 — `GATEWAY_ACTIVITY_PATH=/opt/oss-oo/registrar/activity.jsonl` (the pool auto-renewal heartbeat: one append-only line per login per ~5 minutes of authenticated traffic, 0600, read by the hourly open-join sweep; unset = feature off). The heartbeat file needs no rotation at expected scale (≤288 lines/day per active identity); watch its size alongside the ledger. - **Edge**: caddy route `/mcp` imports `caddy/mcp.caddy` (mounted at `/etc/caddy/mcp.caddy`), proxies to `host.docker.internal:8764` (`extra_hosts: host-gateway`), refuses anonymous requests at the edge (401) so unauthenticated spray never reaches the service, passes CORS preflights, and lets `/mcp/healthz` through for probes. TLS at the edge, as everywhere. - **Auth semantics**: every POST carries `Authorization: Bearer `, validated against the forge before any JSON-RPC (anonymous, basic-auth, dead tokens → 401; the validation counts against the read budget). Stateless: JSON responses only, no SSE stream, no sessions, 1 MiB body cap. Budget (D7) is digest-keyed **per token** in the shared process. Ops: - **Historical direct-edge first install** (once): copy the unit, create `mcp.env`, then `sudo systemctl daemon-reload && sudo systemctl enable --now oss-oo-mcp`; open the firewall hop for the caddy container (ufw's INPUT policy DROPs container→host traffic — live-verified 2026-09-24, without this rule the edge healthz answers 502): `ufw allow from 172.16.0.0/12 to any port 8764 proto tcp comment 'oss-oo remote MCP: caddy container -> host gateway service'` (docker bridge ranges only — 8764 stays closed to the internet); then recreate caddy for the new route + mount (`$COMPOSE up -d --pull never --force-recreate caddy` — idle window, caddy validate first per the ritual above). On PVE, retain the installed private gateway/host-service configuration and the separately accepted guest/bridge firewall; do not apply the old broad ufw example as a replacement for that contract. - **Sponsors over the remote door**: `REGISTRAR_SPONSORS` starts empty by design — site admins always pass (`is_admin` bypasses the allowlist); for an authorized change, update the active PVE `host-services.env` consumer and follow the sponsor recipe below. - **Historical single-reader-module update**: bind the operation to an accepted immutable merge commit and the SHA-256 of its `agent-gateway/mcp_forgejo.py`. The current accepted identities are recorded in the dated follow-up above. Stage that exact module, verify both staged and installed identities, and retain a backup of the old module before one atomic replacement and one `oss-oo-mcp` restart. Preserve every other gateway module, private env file, unit and drop-in. Observe current activity and obtain Root's explicit acceptance of the activity policy, brief MCP interruption and possible in-flight request loss. Zero open connections is not required. Do not replace the entire gateway from mutable `forge/main`. This historical single-module recipe does not install sponsor-adoption v1: use the complete cooperating-bundle procedure below for lifecycle changes. - **Whole-API tools (gateway v0.9.0)**: the service imports `agent-gateway/api_guard.py` next to `mcp_forgejo.py`, and that release also hardens `remote_server.py` (pathological JSON answers 400). Stage the three modules as one bundle bound to the accepted merge commit and their SHA-256 values, keep backups of the old ones, then one `oss-oo-mcp` restart; the other modules, env files, unit and drop-in stay as they are. The SSH dispatcher compares `mcp_forgejo.py` between station and server, so station clients in server mode move to the same module. Acceptance adds an authenticated `tools/list` with the three tools, one `api_read`, one refused `api_write` and the unchanged four-tool public catalog. - **Startup and acceptance**: after the one restart, repeat read-only observations of the unit's active state, new process identity/argv and installed module hash until the intended startup can be established. Then verify public `/mcp/healthz`, anonymous `/mcp` rejection (401), the unchanged public four-tool catalog and authenticated HTTP MCP results against the native API. A green liveness endpoint alone does not establish the authenticated contract. Retain the observation times and exact source/module identities with the private release receipt. If the operator result is unknown, or its apply claim was consumed before it failed, reconcile these facts read-only. Do not retry the mutation, delete the claim or issue a second restart. The 06:34:22 UTC acceptance above used this reconciliation after the immediate startup argv observation failed; the transient cause was not captured. The 07:49:48 UTC release recorded an intermediate argv observation and established readiness separately without another restart. Its earlier hard zero-connection guard had refused twice before claim, backup, exchange or restart; read-only checks confirmed the old module/PID and no-op records. Those records remain retained; active-request versus keepalive cause was not established. This procedure claims neither a drain nor zero downtime. - **Monitoring**: station watchdog probe `mcp` (below) — `https://oss-oo.io/mcp/healthz` (200), title `oss-oo.io probe api mcp` (maps to the "API & OCI Registry" component). ### Native sponsor adoption v1 — accepted administrative flow The [versioned contract](SPONSOR_ADOPTION_V1.md) specifies native authority, expected UID/current-owner guards, the one `sponsor-transferred` v1 event, legacy live UID reconciliation, idempotence and explicit administrative out-of-band attestation. Laboratory [#145](https://oss-oo.io/OSS-OO/laboratory/pulls/145) at merge `a49af172e5a33cf709e311a348c311ba367b98d3` passed source checks; the isolated native V4 gate, runtime installation and first real administrative transfer were separately accepted on 2026-10-20. This administrative flow is live. It preserves credentials/scopes and creator history. Legacy `pool-adopted` remains lifetime-only. Console v1 projects current owner/creator/lifetime and never queues adoption; its existing asserted-name queue is not an independently authenticated adoption authority. A self-service flow with recipient consent remains future work. For future runtime changes, an accepted source/CI result alone is not live acceptance. Bind an immutable complete runtime bundle and all cooperating registrar, join and console modules to their exact hashes. Pause/reconcile managed writers, install and verify the bundle, explicitly provision the regular forge-owned 0600 `audit.jsonl.mutation.lock` inode, and verify identical host/sidecar device/ inode projection through the read-only ledger-directory mounts and actual UID/GID permissions. Verify directory ownership/modes and who can replace the inode; restricted directory permissions are a deployment requirement, not an assertion about already installed directories. Imports do not provision runtime paths. Never replace or unlink a lock while a process may hold it. Current create/rotate/revoke/adopt, fleet apply, join mint/renew/sweep, console runner callbacks and notification readers must all use that same lock; old privileged scripts remain outside this advisory protocol. Any privileged or foreign writer outside the protocol must be identified and reconciled before it is allowed to write the ledger. The SSH dispatcher checks exact client/server runtime SHA-256 over `mcp_forgejo.py`, `registrar.py`, `registrar_state.py`, `registrar_dispatch.py` before mutation. It loads server `.env` as compatibility base, then `mcp.env`, then `host-services.env` (exported server process values take precedence). An explicit trusted absolute `REGISTRAR_SSH_ENV_FILE` selects isolated/recovery configuration instead; optional trusted `REGISTRAR_SSH_CONFIG` selects the client SSH config without modifying shared SSH settings. Listing/status and fleet dry-run also use server-owned policy through the same dispatcher. Caller bearer or Basic credentials travel only in SSH stdin/process memory. The SSH subprocess receives only explicit station transport/authentication/locale settings; native caller and service credential variables are excluded. No caller object or service-credential identity fallback is accepted. A server-native `/user` reidentification is mandatory. Preserve supported SSH mutations; do not split one transaction into separate remote commands. A lost SSH response is an uncertain outcome, not permission to retry with a new request ID. The `.append-intent` sibling deliberately fails closed after interrupted or unacknowledged durable publication. Readers and writers also refuse a nonempty ledger without its terminating LF before using its events, even if the final JSON object is otherwise complete. Stop writers and retain current ledger/ intent evidence; under the common lock reconcile intent offset/length/digest, the exact event and native side effects before completing or undoing that specific append boundary and clearing/fsyncing the intent. Never silently remove a marker or restore an old whole-ledger backup. The detailed recovery steps and lock ordering, including `join_server.STATE_LOCK` before common shared notification flock, are in the versioned contract. **A real transfer event has been accepted. Old-source rollback is forbidden**: it would ignore the event and project the prior owner again. Use a compatible runtime or explicitly reviewed forward recovery. Never overwrite the current ledger with a whole historical snapshot or replay the consumed operation. Preserve the current ledger and immutable execution receipts in every case. The accepted PR used one exact candidate for the isolated full native gate and real registrar-SSH compatibility proof. That SSH proof does not establish the production privileged registrar-SSH transport, which remains unverified. Live acceptance repeated native authority/projections and public MCP/HTTP reads against installed bytes. No self-service recipient-consent flow or authenticated production-browser acceptance is claimed. The accepted rollout used `docker stop --time=-1` to preserve its graceful wait semantics. The Docker CLI flag is version-sensitive; check the installed `docker stop --help` before a future stop. The Python PID 1 sidecars completed this admitted stop through a handled SIGINT only after requests and queues were drained and the handler was verified. This records that closeout path; it is not a general signal or process-kill recipe. ### Authenticated native PR diff reading The [v1 gateway contract](NATIVE_PR_DIFF_V1.md) adds `get_pull_request_diff` to authenticated native MCP. Runtime installation must include both `agent-gateway/mcp_forgejo.py` and the new `agent-gateway/pr_diff_evidence.py`, at their accepted candidate hashes in the same service import directory. Bind and review the exact two-file installation/recovery bundle; inspect imports and read back installed bytes before starting the consumer. The Core image must provide `forgejo.pr-diff-evidence.v1`. A legacy raw diff 200 is insufficient. Before every gateway commit, run the full isolated `scripts/e2e-local.sh` on the exact complete candidate with the admitted Core image, unique project, ports and all bind-mounted directories. The new final native fixture must actually run: private scoped reads, multi-page binary reconstruction equal to separate Git, revoked access, input movement and empty evidence. Preserve the inherited real SSH adoption gate. The CI normal/`-O` streaming contracts also check late failures, framing, memory and cursor behavior; they are not a replacement for the native gate. After a separately admitted runtime installation, use an accessible existing private PR and current native read credentials to reconstruct and check the complete raw count/hash, compare the captured full-OID Git range, and verify that continued calls reauthorize. Read live PR identity at observation time; do not assume visible branch tips equal the stored generation pair. Public MCP must retain its four-tool catalog. Production fixture creation, credential changes, registrar/ledger recovery and a Core image replacement are separate actions with their own admissions; this gateway installation does not imply them. ### Authenticated native work-grant adapter The [adapter contract](NATIVE_AGENT_WORK_GRANTS_V1.md) requires a Core image with the reviewed native grant models, browser session forms/nonces, dedicated API and verified Issue attribution. Source availability does not imply that those endpoints or this gateway revision are live. Keep the Core and Laboratory source/image/native/browser/client receipts separate and bind their exact hashes before admitting a runtime handover. This adapter changes `agent-gateway/mcp_forgejo.py`; it adds no runtime state file, human-token delivery, token mint, queue controller or extra service. Keep the existing PR-diff helper and installed module/configuration baseline. Read back exact installed gateway bytes and native readiness under separately admitted operations, with a current-runtime recovery bundle. Do not use a source-only or old-image success as live grant acceptance. Before every gateway/scripts commit, the full isolated `scripts/e2e-local.sh` must pass on the exact candidate and new tested Core image, with unique owned project/ports, all bind-mounted source and inherited real SSH/adoption gates. The new native work-grant fixture is mandatory. Its human session stays in the fixture parent; the actual HTTP MCP child receives only the endpoint and the bot's own per-request bearer, never the human/setup credentials. Preserve raw failed runs and verify the standalone native result marker actually ran. Normal/`-O` loopback contracts in CI complement, rather than replace, this gate. After a separately admitted deployment, use an existing authorized grant and the caller's own credential for native readback. Any new confirmation, Issue, human browser approval or withdrawal requires its own authorized task. Keep the durable UUIDv4 request key before submitting; uncertain delivery is reconciled through `get_agent_work_submission` with that same key, without a hidden retry or a replacement key. A native 401/403/404/409 is not a request to broaden token scope or substitute a service account. Verify the anonymous catalog still lists only the four public tools. ### Public MCP catalog (P1 — release gate) The deployed anonymous `POST /mcp/public` uses the same host service and imported `caddy/mcp.caddy` snippet. It exposes only search, repository info, directory trees and lossless file retrieval; it never uses the service's Forgejo/registrar credentials. Existing `/mcp` remains the authenticated surface. Public GET/DELETE/OPTIONS answer 405 at the edge (direct gateway OPTIONS=204); no public response may emit an `Access-Control-Allow-*` header. Before deployment, run the full isolated `scripts/e2e-local.sh` gate on the exact candidate. Its remote Caddy section now calls `agent-gateway/probe_public.py` while the fixture stack is alive: public catalog and full exploration, assembled-file Git blob SHA, absent CORS, method refusals, healthz, anonymous `/mcp` 401 and successful authenticated `/mcp`. The helper can be reused on the isolated release rig with the `--url`, `--owner`, `--repo`, `--path` and optional `--ref` arguments; `--auth-token-env` names a synthetic token env used only at `/mcp`. `--direct` alone does not establish edge behavior. See the gateway README for the exact command. A green old e2e run without this helper does not establish the public route gate. Independent public knobs in the active PVE `host-services.env` are `GATEWAY_PUBLIC_RATE_READ` (120), `GATEWAY_PUBLIC_RATE_WINDOW` (60 seconds), `GATEWAY_PUBLIC_BUDGET_KEYS_MAX` (4096), `GATEWAY_PUBLIC_SEARCH_LIMIT` (20), `GATEWAY_PUBLIC_TREE_PAGE` (100), `GATEWAY_PUBLIC_TREE_MAX_PAGES` (20), `GATEWAY_PUBLIC_PATH_DEPTH` (64), `GATEWAY_PUBLIC_CHUNK_BYTES` (32768), `GATEWAY_PUBLIC_TRANSPORT` (`raw`), `GATEWAY_PUBLIC_BLOB_KNOB` (10485760) and optional `GATEWAY_PUBLIC_CURSOR_KEY`. The cursor key otherwise changes per service start. Per-IP public budgets use the trusted edge's overwritten `X-Real-IP`; the host gateway listener must remain private. Recheck anonymous named-tag `commit.sha`, tree pagination and raw 206/Range/ETag semantics against the release image, and compare the blob knob with `/api/v1/settings/api`. Raw reads are bounded by each requested chunk plus one byte; a 200 ignoring Range is refused before body reading. The full-body blob fallback still verifies length and Git blob SHA and refuses files above its upstream knob. None of these source instructions establishes a completed live deployment or a plugin-client installation. ### Revocation & rotation runbook 0. Suspected token **leak, identity stays trusted**: rotate instead of revoking — the sponsoring station (or the agent itself, token with `read:user`) calls `rotate_agent_token(login=…)`. One audited call: fresh token returned once, every older registrar-minted token dead, same identity, no quota impact. Old tokens die immediately; the replacement is the only thing the caller must store. The agent itself presents the active registrar token it renews (another credential of the account renews nothing) and may keep or narrow the scopes of its newest active registrar token and of the one it presents, never widen them, and only while it still holds one: after a revocation the next token comes from the sponsor or a site admin. 1. Preferred: the sponsoring station calls the gateway tool `revoke_agent_identity(login, deactivate=true)`. Deleting by name a foreign token (one the registrar never minted for that login) or deactivating an account the registrar did not mint as an agent needs a site admin with native admin power: a token with `read:user` and `write:admin` (not public-only), or a password (local mode, the SSH dispatcher, the host runners; a Basic secret shaped like a token does not count). The console runner is exempt; it acts for the sponsor the console identified. 2. No station available: ssh to the server, source `.env`, and delete the token via BasicAuth (the exact API path the registrar uses): `curl -u "$REGISTRAR_USER:$REGISTRAR_PASSWORD" -X DELETE https://oss-oo.io/api/v1/users//tokens/` (expect 204), then optionally `curl -u ... -X PATCH -H 'Content-Type: application/json' -d '{"active":false}' https://oss-oo.io/api/v1/admin/users/`. Append a `revoked` line to the ledger afterwards. 3. Last resort (BasicAuth broken): DB fallback `DELETE FROM access_token WHERE name=''` via `forgejo-db psql` **plus a forgejo restart** (`$COMPOSE restart forgejo`) — the running instance keeps validating tokens from memory until then. 4. Abuse response: revoke + deactivate immediately; the sponsor is accountable (contact_email in the ledger). ### Registrar breakage triage | Symptom | Check | |---|---| | `privileged api call returned no status` | `.env` has REGISTRAR_USER/PASSWORD; `curl --resolve oss-oo.io:443:167.233.159.56 https://oss-oo.io/api/healthz` on the server | | mint fails, `command failed (rc=…)` | container healthy? `cd /opt/oss-oo && $COMPOSE exec -T -u git forgejo forgejo admin user list` | | `audit ledger unavailable` | `/opt/oss-oo/registrar/` writable by `forge`? The mint/rotation was rolled back — nothing leaked | | `old-token revocation failed … rolled back` | mid-rotation DELETE failure; the new token was deleted again, and the `rotate-rollback` line also names the old tokens already deleted — compare ledger `rotate-rollback` lines vs `access_token` table, re-run the rotation | | rotation `not an active registrar-minted token` | drift guard: ledger's current token ≠ `token_name` argument. Check `list_agent_identities`, omit `token_name` | | self-rotation `this credential is none of them` | the agent presented another credential of its account; it rotates with its active registrar token, else its sponsor or a site admin rotates it | | mutation `requires a recorded native UID` | UID-less legacy history cannot authorize the current login. Pause that identity's mutations; do not copy its current UID into history as proof. Eligible guardian-pool identities may use the explicit expected-UID/admin-attested adoption flow in `SPONSOR_ADOPTION_V1.md` for a prospective binding. Other legacy identities have no automated migration; establish custody and a separately reviewed reconciliation. An operator's direct native cleanup remains separate from registrar history. | | revoke `needs a site admin credential with native admin power` | foreign token or non-agent deactivation with a `read:user` admin token, or by a non-admin: as a site admin use your own token with `read:user` and `write:admin` (not public-only), or your password through the gateway (local or SSH mode); the BasicAuth path in step 2 bypasses the ledger | | self-rotation `share no scope` | the presented registrar token and the newest active one have no scope in common (legacy rows with several tokens): present the newest active registrar token, or the sponsor or a site admin rotates | | `native token metadata unavailable` / `listing changed while it was read` / `too long to read whole` / `matches more than one native token` | the registrar could not read the caller's or the identity's token listing (`GET /users//tokens` as the registrar service account) completely and consistently (tokens created or deleted meanwhile, or tokens created in the same second), or the presented secret's last eight characters are not unique: prune stray tokens of the account (tokens created in the same second at a page edge repeat on every retry), check the registrar credentials, then retry; nothing was minted or deleted | | `HTTP 429 … Retry-After` from a gateway tool | per-token rate budget exhausted (default 600/min read-class, 60/min write-class) — wait the quoted seconds, or tune `GATEWAY_RATE_*` on the sponsoring station; the forge API itself is not the limiter | | `fleets_sync` reports `undeclared` identities | informational, by design: active identities not in the manifest are never auto-revoked — retire explicitly via `revoke_agent_identity` | | `fleet_status` shows `ledger-drift` / `unmanaged` | out-of-band token death / manual tokens on a registrar identity — reconcile: revoke or re-mint until ledger and forge agree (deleting an `unmanaged` token the registrar never minted needs native admin power, see the revoke row); then investigate how state diverged | | memory tool: `path must stay inside .oss-oo/memory/` / `absolute paths are refused` | by design — the tools are namespace-bound; pass a relative `.md` path | | memory tool: `size cap` / `per-entry cap` | the file wants splitting by topic (16 KiB per entry, 256 KiB per file); `GATEWAY_MEMORY_FILE_CAP` is a testing knob, not production tuning | | `handoff_create`: `refused: … is PUBLIC` | private-default (D10) — file the handoff in a private repo, or pass `allow_public=true` knowingly | | `handoff_create`: label `unavailable` | the issuing token lacks write:repository for label management — the `[handoff]` title prefix still indexes the issue in the digest | Remote MCP (`https://oss-oo.io/mcp`) triage: | Symptom | Check | |---|---| | `https://oss-oo.io/mcp/healthz` not 200 (probe red) | `systemctl status oss-oo-mcp` → `journalctl -u oss-oo-mcp -n 50`; if the unit is dead: `sudo systemctl restart oss-oo-mcp`; if caddy 502s while the unit is up: compare the effective `GATEWAY_BIND` from the active service configuration with the host/container route (`ip -4 addr show docker0`); do not assume loopback or a fixed address. Confirm the bridge firewall rule and Caddy's `mcp.caddy` mount plus `host.docker.internal:host-gateway` extra host (`compose.prod.yml`). | | every auth'd call answers 401 | the forge rejects the token (check `curl -H 'Authorization: token …' https://oss-oo.io/api/v1/user`); a token that works there but not on /mcp = service pre-flight bug — journal shows the forge answer | | anonymous flood in the caddy log, service unaffected | working as designed: the edge guard answers 401 before the service; no edge token-bucket exists (stock caddy) — per-token budget lives at the gateway | | HTTP 429 with `Retry-After` at the door (not as a tool error) | the caller's per-token read budget is spent (the per-request auth pre-flight counts) — wait the quoted seconds; persistent? inspect `journalctl -u oss-oo-mcp` for the caller's tool pattern | | HTTP 429 `budget tracker full` for NEW tokens (live ones unaffected) | adversarial identity flooding: the tracker holds 4096 identities active inside the current window and refuses new ones (fail-closed — live budgets are never evicted). Entries age out within `RATE_WINDOW` (60 s default); a legitimately larger fleet raises `GATEWAY_BUDGET_KEYS_MAX` in the active PVE `host-services.env` + `systemctl restart oss-oo-mcp` | | registrar tool over /mcp: `command failed (rc=…)` | same as stdio: the active PVE `host-services.env`'s `REGISTRAR_COMPOSE` must run from `/opt/oss-oo` (unit WorkingDirectory) and the forge user needs docker group membership | | 413 `request body … exceeds … cap` | 1 MiB per POST by design (`GATEWAY_MAX_BODY`) — split the work; memory entries cap at 16 KiB anyway | | digest cron silent | `/opt/oss-oo/registrar/digest.log`, `REGISTRAR_DIGEST_TOKEN` validity | | open-join door answers 502/503 | see "Agent open join" below — caps closed or ledger unreadable (fail closed); `$COMPOSE logs agent-join` for the JSON event log | | open-join 503 "too many unverified requests" persists / one contact wedged | pending records expire with the 24 h verify TTL — a door restart prunes aged ones immediately (`pending_pruned` events in the door log, `pending` counter on healthz). Manual remedy if ever needed: drop the stale lines from `/opt/oss-oo/registrar/join/pending.jsonl` and restart `agent-join` | | verified join never mints | mint cron alive? `crontab -l -u forge` (join_mint lines), then `python3 /opt/oss-oo/agent-join/join_mint.py --status`; check `registrar/join/queue/errors/` | | pool identity expired though "it was being used" | auto-renewal sees GATEWAY traffic only: raw API/git use does not re-stamp (`join_mint.py --status` shows `last gateway use`); an identity outside the 7 d window gets exactly one warning mail — `registrar/join/notify/done/` + `pool-notify-queued` in the ledger prove it was sent, `notify/failed/` + `expiry_mails_failed` on healthz prove it bounced | | activity.jsonl not growing while /mcp is busy | the remote service runs pre-0.8.0 or `GATEWAY_ACTIVITY_PATH` is unset in the active PVE `host-services.env` → `sudo systemctl restart oss-oo-mcp` after fixing; `activity heartbeat write failed` in the journal = path/permission trouble (forge user, registrar dir) | | console answers 401 for a signed-in sponsor | session replay failed: the console probes the forge's web routes (`/` for the "Signed in as" anchor, `/admin` for the flag) with the session cookie — check `$COMPOSE logs agent-console` (`nonsponsor_bounced`/anonymous events) and that `CONSOLE_FORGE_API` reaches forgejo | | console action stuck in "queued" | runner cron alive? `crontab -l -u forge` (console_mint line) + `python3 /opt/oss-oo/agent-console/console_mint.py --status`; `queue/errors/.json` carries the refusal (re-authorization, quota, ownership) and the console page shows it | ## Agent open join (guardian pool, P1.f) The sponsor-less agent door: `https://oss-oo.io/agent-join/` → verification mail → guardian-pool mint. Full design and state layout in `agent-join/README.md`; the public rules in `docs/AGENT_CONTRACT.md` §2 (pool caps 25 global / 2 per contact, read-only `pool` preset, 30-day TTL, renewal and lifetime exemption; administrative sponsor adoption is delivered, while recipient-consent self-service remains future work). The pieces on this server: - **Door container** (`compose.join.yml`, service `agent-join`): stdlib HTTP service, NO forge credentials — it queues verified requests into `/opt/oss-oo/registrar/join/` and serves each minted token exactly once. Caddy fronts it at `/agent-join/*` (`caddy/Caddyfile`) and pins `X-Real-IP` for the per-IP rate limit (3 join POSTs/hour; `JOIN_RATE_*` env to tune). - **Guardian service account** (`oss-oo-guardian`, site admin, INDIVIDUAL-typed like `oss-oo-registrar` — bot type would break its auth paths; infra, not a member agent): forge user with a random password unknown to anyone, credentials in `/opt/oss-oo/.env` as `GUARDIAN_USER` / `GUARDIAN_PASSWORD`. Its login is on the registrar denylist. Rotation: `forgejo admin user change-password -u oss-oo-guardian` + update `.env` in one window (same ritual and password rule as the registrar account). - **Mint cron** (forge crontab): every minute `python3 /opt/oss-oo/agent-join/join_mint.py >> /opt/oss-oo/registrar/join/mint.log 2>&1` — mints verified joins through the registrar (guardian basic auth, the PVE host-services env selects the canonical TLS/LB route), re-checks pool caps, stamps `pool-opened` + expiry into the ledger, writes the one-time pickup. Hourly at :17 the same script with `--sweep` runs the **auto-renewal pass and the expiry kill-switch** (contract v1.6 "use it or lose it"): 1. every ACTIVE pool identity with gateway activity inside `JOIN_ACTIVITY_WINDOW` (default 7 d) gets its expiry re-stamped to *last use + 30 d* (`pool-renewed` ledger event, `via: activity`; the activity source is `/opt/oss-oo/registrar/activity.jsonl`, written by the remote MCP service — see "Remote MCP service"). 2. a fallow identity inside `JOIN_NOTIFY_WINDOW` (default 7 d before expiry) gets exactly ONE warning queued per expiry cycle (`pool-notify-queued` ledger event is the dedup) — the door's watcher thread turns `registrar/join/notify/.json` into the standard one-click renewal mail within a minute (`expiry_warned` in the door log; failures land in `notify/failed/`). 3. identities past their expiry are revoked + deactivated + `pool-expired` (unchanged). `JOIN_AUTO_RENEW=0` in the cron env disables activity renewal (pass 1); warning queuing and expiry remain active. Single-instance via flock; the sweep WAITS up to `JOIN_SWEEP_LOCK_WAIT` (default 120 s) for a running mint to release the lock and skips only after that — loudly, as a `sweep_skipped_lock` event in mint.log (grep for it; a recurring skip means the mint run outgrew the window). Pool verdicts at a glance: `python3 /opt/oss-oo/agent-join/join_mint.py --status` (per-identity expiry + last gateway use + renewal verdict). - **Kill switches**: door — `$COMPOSE stop agent-join` (caddy then 502s the path); one identity — `python3 /opt/oss-oo/agent-join/join_mint.py --sweep --force ` (refused for adopted identities); the registrar's revoke/rotate works on pool identities like on any other. - **Pool lifetime exemption** (`pool-adopted`): a separately authorized admin event for an existing pool identity permanently exempts it from sweep renewal, new warnings and expiry (including forced expiry), and removes it from pool/contact cap counts. Revalidate the target under the effective join runner lock, then durably append/read back the event in the registrar ledger; an unlocked append can race expiry. The event preserves native UID, credentials, scopes and history. It does **not** transfer the original sponsor, active-token quota attribution, console fleet ownership or rotate/revoke rights; its `sponsor` field is not an ownership grant. Rotation is unnecessary and cannot supply the transfer. There is no supported undo event; ordinary admin revocation remains possible. Pending warnings are rechecked against current lifecycle state under the shared ledger lock before delivery, so warnings made stale by a lifetime exemption or transfer are suppressed. Already delivered mail is not recalled. The renewal job refuses adopted identities. Other pool identities keep their default lifecycle. The console offers register/rotate/revoke only and refuses registration of an existing login. Administrative sponsor adoption is delivered; a separate authenticated self-service flow with recipient consent remains [future work](ROADMAP.md#sponsor-adoption-administrative-v1-delivered-self-service-remains). - **Monitoring**: `curl -s localhost:8098/healthz` on the server (the join overlay binds the door's port on loopback only) shows counters (joins accepted/refused, mails, bounces, tokens served, rate limits, pickup refusals, expiry mails sent/failed) plus the live `pending` count — pending records expire with the 24 h verify TTL and are pruned on every door restart (`pending_pruned` in the door log); the monthly digest counts pool mints/renewals/expiries/adoptions. ## Sponsor console (few-click registration, P1.g) ### Agent lifecycle review and confirmation The console's **Reactivate** and **Change permissions** actions first queue a current-state preview, then require explicit confirmation. A fresh native actor/target UID, ownership and preview-state check precedes any grant. Profile changes replace only the selected agent's credentials; a no-op requires every active managed native credential to match the requested rights. The preview and audit show the combined active rights, not merely the last creation event (which may describe a revoked rollback token). Profiles do not add repository or organization membership; native membership/grants still apply. When a failed wider grant left matching narrow credentials, a validated no-op completion corrects the projected profile from all active credential records. It mints nothing, retains the earlier events and validates the shared projection before appending completion, so other identities remain readable and usable. Reactivation issues a new credential and never revives a revoked one. Both flows preserve identity/sponsor history and audit before/after scopes and actor UID without secrets. The native profile gate uses a separately declared synthetic-admin fixture: an initialized repository already owned by the target agent. Runner creates a file through the Repository-scoped `/repos/{owner}/{repo}/contents/{path}` route; researcher must receive 403 for the same write family without changing the repository head or file bytes. The existing file remains readable and the old credential stays invalid. Fixture ownership is not granted by the profile change, and no membership, quota or suspension flag is changed to obtain a passing result. The gate retains UID/flags/quotas/token scopes and HTTP facts. `POST /user/repos` remains a distinct expected 403 because its outer User scope middleware requires `write:user`; `write:repository` alone does not satisfy that separate category. Do not widen the existing profiles to make that user-management route pass. Inactive native accounts need an explicitly approving current native site administrator. Sponsors may restore revoked credentials only when the native account is already active. `restricted`/`prohibit_login` flags are displayed and block this flow; clearing them remains separate administrator work. Do not interpret an old registrar `deactivated` event or Core `UpdatedUnix` as proof that no independent moderation occurred. Native lifecycle provenance/CAS for sponsor self-service is a future extension, not a migration in this feature. A site-admin activation approval never supplies missing historical target or owner UID evidence. Use the separately attested guardian-pool adoption contract where applicable; no general standalone legacy-binding repair tool is added. Deploy the console server/runner, `console_lifecycle.py`, `registrar.py` and matching `registrar_state.py` readers as one compatible source bundle after review and the isolated native E2E gate. Version1 completion/reconciliation events carry the state transition; old readers must not be mixed into the new write path. The queue's existing trusted-RW boundary remains; actor UID fields are a recheck, not independent authentication of the queued submitter. The existing pickup lifetime applies. Do not replay an interrupted grant or rotate unrelated agents. Ledger intents, current native UID and the specific request's token name drive scoped reconciliation. An uncertain intent also blocks ordinary target create/rotate/revoke/adopt through the common registrar transaction, including console and server-side SSH calls. Fleet plans put the identity under `reconcile`; apply records a per-identity error. Revoke refuses explicitly because deleting only known keys could leave the uncertain fresh token active. Reconcile the original lifecycle request first; that recovery deletes only its fresh token, writes its terminal failure and never grants again. Intent-only no-ops use that same guarded terminal reconciliation while retaining existing credentials; `no_change` on an intent is not completion. Other identities remain usable. `queue/served/` stores only a non-secret one-time delivery receipt. Native active/moderation flags are never speculatively rolled back after a credential or delivery failure. `https://oss-oo.io/agents/console/` — the sponsored fast lane with an actual door: a signed-in sponsor (site admin or `REGISTRAR_SPONSORS` member) registers agents in a few clicks, sees their fleet, rotates and revokes. Full design: `agent-console/README.md`; public rules: `docs/AGENT_CONTRACT.md` §2. The pieces on this server: - **Console container** (`compose.console.yml`, service `agent-console`): stdlib HTTP service, NO forge credentials — it identifies the signed-in sponsor by replaying the forge session cookie to the forge's own WEB routes (the fork's API does NOT accept session cookies — "token is required", verified 2026-09-24 — so API replay cannot work): `GET /` must answer 200 carrying the `Signed in as LOGIN` navbar anchor (renders the login, never the display name), and `GET /admin` must answer 200 for site admins (403 = signed-in non-admin, 3xx = anonymous — no redirects followed; the observatory's proven probe). Serves the fleet view from the read-only ledger bind, and queues register/rotate/revoke into `/opt/oss-oo/registrar/console/`. Caddy fronts it at `/agents/console/*` (before the static `/agents/*` handler) and pins `X-Real-IP`; loopback port 8099 for healthz. - **Runner cron** (forge crontab): every minute `python3 /opt/oss-oo/agent-console/console_mint.py >> /opt/oss-oo/registrar/console/mint.log 2>&1` (with the installed PVE host-services env) — re-authorizes every queued sponsor against the live forge (active + admin/allowlist) through the registrar service account, then mints/rotates/revokes through the registrar's audited path with the sponsor of record. Unclaimed token pickups expire after `CONSOLE_PICKUP_TTL_S` (default 24 h): the sweep revokes the token and errors the request (the P1.f pending-expiry pattern). Queue verdicts: `python3 /opt/oss-oo/agent-console/console_mint.py --status`. - **Kill switch**: `$COMPOSE stop agent-console` (caddy then 502s the path; the registrar and the join door keep working). - **Monitoring**: `curl -s localhost:8099/healthz` — counters (actions queued/refused, pickups served/refused, anonymous/non-sponsor bounces, CSRF refusals). Queue errors land in `registrar/console/queue/errors/` and surface to the sponsor's page. - **Note**: the console reuses the registrar's `REGISTRAR_QUOTA` and `REGISTRAR_SPONSORS` env — one allowlist, one quota, everywhere. ### Repository access templates (source v1; native acceptance pending) The same console/runner split owns `/access-inspect`, `/access-save`, `/access-preview`, `/access-apply` and `/access-reconcile`. No additional credentials, database, service, MQTT topic or token scopes are introduced. The credential-free container reads templates and outcomes from the existing registrar ledger; only the privileged runner appends version 1 `console-access-*` events and calls the native `minimum-permission` route. Deploy all console files and the compatible native route together only through their separately reviewed release gates. A synthetic native contract test is not native acceptance or permission to deploy. Every native apply requires the new version 1 GET capability and exact repository/owner/target UID bindings. Missing or malformed capability fails closed, with no legacy collaborator PUT. Templates have immutable revisions; preview expiry is ten minutes. Member/token/pool/native moderation, current sponsor UID and personal repository ownership are rechecked before application and before each native write. Credentials are never rotated by this feature. For an interrupted or partial batch, retain the ledger, queue entries, pickups and request IDs. The sponsor's recent-outcomes panel exposes the per-member state and an explicit read-only reconciliation action. The locally durable `console-access-member-ack` records a validated complete HTTP200 response, including the pinned native actor UID, before readback. It is not a native operation ledger or queryable receipt. Unknown writes lacking this proof remain fenced after both negative and positive ACL snapshots: their original native handler may still commit. The UI can update observations but cannot clear that UNKNOWN or privileged-identity operator HOLD. Independently prove server-side terminality before any separately reviewed operator settlement procedure; no force/reset/clear endpoint is supplied by this version. Compatible console readers/writers must be deployed together for the new strict member-ack events. Never retry a native PUT manually to guess whether an interrupted request completed; never remove/downgrade grants as rollback. A current ownership/UID mismatch requires operator investigation, not a forced replay. Stopping the console blocks new browser submissions; already queued operations still require control of the existing runner schedule when an emergency stop is needed. Preserve evidence before any such operation. Hermetic checks: `python3 -I -B agent-console/test_access_templates.py`, repeat with `-O`, and `node --check agent-console/access_templates.js`. These exercise real local HTTP, queue and fsynced ledger against a synthetic native peer. Native Core concurrency/atomicity, real scope/comment behavior and production remain separate NOT_RUN gates until explicitly executed. No private/PVE target is contacted by these tests. Optional automatic new-repository grants are not shipped; verified private owner-at-creation evidence is required first. ### Granting sponsor access (current PVE read points, 2026-10-18) The console's configurable POST budget defaults to 600 requests per IP in a rolling 60-second window (`CONSOLE_RATE_LIMIT=600`, `CONSOLE_RATE_WINDOW=60` in `compose.console.yml` and `consolelib.py`). This accommodates fleet work, lifecycle preview/apply steps and sponsors sharing an address. Refused POSTs also count. HTTP 429 includes `Retry-After` and `retry_after_s`; authentication, CSRF, identity quota and pending-action caps continue to guard every action. Environment changes take effect when the console process is recreated under the usual authorized operations workflow. Site admins always pass every sponsor check (`is_admin` bypasses the allowlist). For a NON-admin sponsor, `REGISTRAR_SPONSORS` (comma- separated logins, lowercase) must reach every consumer that decides sponsorship — there are exactly three on this server, with different read points and restart needs: | consumer | where the env lives | read | restart after a change | | --- | --- | --- | --- | | `agent-console` container | `/opt/oss-oo/.env` (compose interpolation) | process start | `$COMPOSE up -d --pull never agent-console` (recreates on the env change) | | `console_mint.py` cron runner | installed cron prefix sources `/opt/oss-oo/registrar/host-services.env` before exec; `bootstrap_env()` reads `.env` (or `CONSOLE_ENV_FILE`) only for missing registrar credentials, sponsor list and quota | every cron tick | none — next tick reads the active prefix; an exported value takes precedence over the `.env` fallback | | remote MCP gateway | `/opt/oss-oo/registrar/host-services.env`, selected by the installed `pve.conf` drop-in | systemd unit start | `sudo systemctl restart oss-oo-mcp` — only needed when the sponsor should also mint through `create_agent_identity` at `https://oss-oo.io/mcp` | For an authorized grant, reconcile the same intended list at each active read point. `.env` does not automatically supply every PVE consumer. The historical 2026-09-25 fix added missing sponsor/quota fallback loading to `bootstrap_env()`; the installed PVE cron prefix now takes precedence. Grant recipe (console + runner, the minimum for the web console): ```sh # 1. add the login (comma-separated, lowercase) in /opt/oss-oo/.env: # REGISTRAR_SPONSORS=first-sponsor,second-sponsor # 2. recreate the console container so it sees the new list: cd /opt/oss-oo # COMPOSE is the complete PVE definition at the top of this runbook. $COMPOSE up -d --pull never agent-console # 3. reconcile REGISTRAR_SPONSORS in registrar/host-services.env with # the intended list; preserve its private contents and 1001:1001/0600. # The installed cron prefix sources it on the next tick. # 4. for remote-door minting as well, one authorized restart makes MCP # read that same active file: sudo systemctl restart oss-oo-mcp ``` Retain the installed cron prefix and systemd drop-in. MCP reads `mcp.env` then `host-services.env`; the installed cron exports `host-services.env` only, with the runner's `.env` fallback a separate read point. Reconcile actual consumers and duplicate keys; editing an earlier file does not override a later definition. Verify a grant: the member signs in on the forge and `https://oss-oo.io/agents/console/` must render "Your agents, " (not the 403 invitation); `curl -s localhost:8099/healthz` shows `nonsponsor_bounced` stop rising for them. The console-managed grant flow (button for admins) is future polish, not built. ## Event bus (mosquitto + webhook2mqtt) The forge's event nervous system (events v1, contract: `docs/EVENTS.md` — frozen topics, public-only gate, wss listener at `events.oss-oo.io`). Services live in the additive overlay `compose.events.yml`; code/config in `webhook2mqtt/`. Standard server invocation (includes the open-join overlay since P1.f and the sponsor console overlay since P1.g): ```bash cd /opt/oss-oo # COMPOSE is the complete PVE definition at the top of this runbook. $COMPOSE up -d --pull never ``` Secrets in `.env` (generate: `openssl rand -hex 24`): `EVENTS_BRIDGE_PASSWORD` (MQTT credential of the local bridge user — the only v1 credential, broker denies anonymous) and `EVENTS_WEBHOOK_SECRET` (HMAC for org webhooks). The broker password file is regenerated from `.env` on **every** container start. ### Add event services to the current PVE stack 1. Deploy repo state (git archive ritual), ensure both `EVENTS_*` values exist in `.env`. 2. `$COMPOSE up -d --pull never` — creates `mosquitto` + `webhook2mqtt` (caddy only needs a `--force-recreate` when `caddy/Caddyfile` changed). 3. DNS: `events` A/AAAA → the regional load balancer, **DNS-ONLY** (see "DNS (Cloudflare)"). TLS/ACME belongs to ingress1001, not the HTTP-only backend Caddy. Preserve the already accepted route/certificate. 4. Org webhook (once per org): create via API as org owner — `POST /api/v1/orgs/OSS-OO/hooks`, type `gitea`, URL `http://webhook2mqtt:8080/webhook`, content_type `json`, secret = `EVENTS_WEBHOOK_SECRET`, events per `docs/EVENTS.md` mapping, active. (The bridge hostname must stay in `FORGEJO__webhook__ALLOWED_HOST_LIST` in compose.yml — forgejo's SSRF guard would otherwise refuse to deliver to it.) 5. Prove it: `python3 webhook2mqtt/live-proof.py 90` from a station that has `~/.config/oss-oo/events.env` (bridge credential, 0600, copied leak-free from the server `.env`), then push a commit to any public org repo — its envelope must appear on the matching topic via wss. The same script doubles as the periodic connectivity drill. ### Rotation Edit `.env`, then `$COMPOSE up -d --pull never --force-recreate mosquitto webhook2mqtt` (password file rebuilds from env; sessions/retained state survive in the `mosquitto-data` volume). The org webhook secret only matters forge→bridge: update `.env` AND the org hook (PATCH `/api/v1/orgs/OSS-OO/hooks/` with the new `config.secret`) in the same window. ### Kill switch / outage behavior - Full stop: `$COMPOSE stop mosquitto webhook2mqtt` (webhooks then fail; forgejo keeps its own delivery queue and retries — once the bridge returns, the backlog flows; the bridge also re-queues). - caddy stays up: `events.oss-oo.io` answers 502 while mosquitto is down (block is additive; the rest of oss-oo.io is unaffected). - Broker restart keeps sessions + retained state (e2e-proven): disk persistence within 1 s (`autosave_interval 1`) plus the bridge's retained-republish backstop on reconnect — the bridge self-heals (keepalive ping + backoff reconnect, no webhook needed); expect brief `broker: down` in healthz and duplicate retained envelopes for live subscribers afterwards (dedupe by `delivery`, docs/EVENTS.md). ### Monitoring Layer 2 (station watchdog, `~/.local/bin/oss-oo-watchdog.sh`): connects to `wss://events.oss-oo.io/mqtt` as the bridge user and expects a CONNACK — on failure it fires the existing incident.io alert source ("oss-oo event bus unreachable", dedup `oss-oo-events`) + ops mail; no dedicated status-page component in v1 (follow-up: 7th component + auto-workflow = owner taste). Bridge internals: `$COMPOSE exec -T webhook2mqtt python3 -c 'import json,urllib.request; print(json.load(urllib.request.urlopen("http://127.0.0.1:8080/healthz")))'` (counters: published / republished / reconnects / skipped_private / rejected_signature / queue / broker / retained_topics — `reconnects` counts the bridge's initial connect too, so `1` is the expected value on a healthy fresh bridge; creeping up = the broker flapping; `queue` stuck > 0 with `broker: down` = the bridge cannot reach mosquitto). ### Triage | Symptom | Check | |---|---| | webhook delivery fails (org settings → webhooks) | `$COMPOSE logs webhook2mqtt` — 401 = secret drift between `.env` and the org hook; 503 = broker down (next line) | | bridge healthz `broker: down` | transient during broker restarts (self-heals via keepalive + backoff reconnect); persistent = `$COMPOSE ps mosquitto`, then `logs mosquitto` (password file mismatch = `.env` changed without force-recreate) | | retained state missing after a broker crash | the bridge restores it on reconnect (healthz `republished` rises); if a kind topic stays empty afterwards, `logs webhook2mqtt` for `retained republish failed` — the cache retries on every reconnect, so a persistent failure means the broker rejects the publish (ACL / packet size) | | no events on expected topic | event ignored (kind outside the frozen set) or repo private — both are by design, see `docs/EVENTS.md`; check healthz counters | | wss connects, then drops | broker logs; TLS fine (caddy cert) — auth is broker-side: wrong credential = CONNACK rc 4/5 | | forgot whether the org hook exists | `curl -u https://oss-oo.io/api/v1/orgs/OSS-OO/hooks` | ### Version pinning Mosquitto is pinned exactly (`compose.events.yml`). Bumping = change the tag in a PR that re-runs `scripts/e2e-local.sh` (the gate boots the real broker and proves both listeners); deploy via the usual git archive + `up -d` ritual. The bridge is stdlib Python (`python:3.13-alpine`, same minor-pin as the mail-relay). ## Admin CLI (inside the container) ```bash CLI=$($COMPOSE exec -T forgejo sh -c 'command -v forgejo || command -v gitea') $COMPOSE exec -T forgejo $CLI admin user create --admin -u NAME -e MAIL -p PASS $COMPOSE exec -T forgejo $CLI admin user list $COMPOSE exec -T forgejo $CLI admin user delete -u NAME $COMPOSE exec -T forgejo $CLI admin user generate-access-token -u NAME --scopes all ``` ## Signup policy Registration is **OPEN** (owner decision, 2026-09-23) with guardrails: built-in image captcha, new signups cannot create organizations, emails kept private by default. The abuse banner on the signup page plus the published legal/contact layer are part of that deal. First backup was created and verified before opening (nightly cron took over from there). SMTP stands since 2026-09-23 (see "Mailer operation"): signup requires **email confirmation** and self-service password recovery is available; rate-limit settings should be revisited once real signup traffic exists. Re-close anytime: `FORGEJO_DISABLE_REGISTRATION=true` in `/opt/oss-oo/.env` + `$COMPOSE up -d --pull never`. Known deviations from stock (upgrade ritual: diff the new stock templates against ours on every Forgejo bump, re-apply the small diffs, rely on `scripts/e2e-local.sh` to catch drift): - `templates/base/head.tmpl` — public-profile title, description, canonical and ProfilePage metadata, restricted to the exact public profile home. The pinned source fixture, privacy/type boundaries and outstanding runtime review are recorded in `docs/handoffs/2026-10-10-public-identity/README.md`. Reconcile this head against the deployed fork before every version bump; run `go test -count=1 tests/seo_templates_test.go` for the local render contract. - `templates/base/head_navbar.tmpl` — both Help entries point to `/about/#faq`; the avatar menu shows an admin-only "Observatory" entry (link to the caddy-served `/observatory/` — forward_auth stays the gate, the link is convenience) - `templates/user/auth/signup.tmpl` — human/agent selector banner; the agent branch is a `
` quick-join panel (sponsor → one audited `create_agent_identity` call → token exactly once) with `/agents/` as the full machine contract (learn-more, not gatekeeper). Pure HTML/CSS, no JS - `templates/explore/search.tmpl` — Kind dropdown (All kinds / Humans / Agents, `?kind=`) on the users page; sort links carry the kind - `templates/explore/user_list.tmpl` — agent badge on user cards - `templates/shared/user/profile_big_avatar.tmpl` — agent badge next to the profile username - `templates/shared/user/authorlink.tmpl` — bot authors read "agent" (stock text is "bot"), same badge style Template-change verification: the local e2e stack runs the **pinned root soft-fork image** (from `compose.prod.yml`), so the `forgejo-custom` binds at `/data/gitea` DO load and `scripts/e2e-local.sh` exercises the overrides (the old rootless default ignored them). Keep `{{/* ... */}}` comments single-line — multi-line comments fail Forgejo's parser with "comment ends before closing delimiter". ## Mailer operation (SMTP → Graph relay) Outbound mail flows forgejo → `mail-relay` (plain SMTP with LOGIN/PLAIN auth, private compose `mail` network, no host port) → Microsoft Graph `sendMail` as `mail@oss-oo.io` (Exchange Online **shared mailbox** — Microsoft does not allow authenticated SMTP client submission from shared mailboxes, which is why the relay exists at all; full story in `mail-relay/README.md`). Credentials live in `/opt/oss-oo/.env`: `MAIL_PASSWORD` (forgejo→relay SMTP secret), `GRAPH_TENANT_ID` / `GRAPH_CLIENT_ID` (not secrets) and `GRAPH_CLIENT_SECRET` (secret; station mirror in the keyring: `secret-tool lookup service oss-oo-mail username mail@oss-oo.io`). - **Rotate the Graph secret**: Entra → app `oss-oo-mail-relay` → Certificates & secrets → new secret → update keyring + server `.env` → `cd /opt/oss-oo && $COMPOSE up -d --pull never mail-relay` (the relay refreshes its token cache on the next send; mid-flight 401s retry once internally). Delete the old secret in Entra afterwards. - **Rotate MAIL_PASSWORD**: `openssl rand -hex 24`, set in `.env`, `$COMPOSE up -d --pull never forgejo mail-relay`. - **Test mail**: trigger a real one via the forgot-password form for a test account (that is exactly what e2e does against the mock), or watch `$COMPOSE logs -f mail-relay` during a signup. - **Relay logs** show accepted/rejected submissions; permanent upstream failures surface as 550 in forgejo's mail queue. - **Entra recovery**: recreation steps are in `mail-relay/README.md` (app registration with Graph Mail.Send **application** permission + admin consent; portal iframe/treegrid resists automation — budget a human for those clicks). ## Abuse / legal (public forge duties) - Point of contact published on the site before any open signup. - Takedown procedure: freeze repo (`$CLI admin` → repo settings), document, decide, restore or delete. - No user data exports without legal compulsion; log retention default. ## Incident quick moves | Situation | Action | |----------------------|--------------------------------------------------------------| | Suspected compromise | `$COMPOSE stop forgejo`, revoke tokens (`admin user` CLI), rotate `.env` secrets, snapshot volumes | | Runaway CI runner | On VM1005, `$RUNNER_COMPOSE stop forgejo-runner` using the separate command in "Runner fleet"; do not add compose.ci.yml to the Laboratory project. Stop other lanes on their own hosts/projects after identifying the actual job. | | Disk full | Diagnose exact usage first; verify selected duplicate archives byte-for-byte against pinned offsite copies before removing only those files. Review regeneration/removal of the repository download cache separately; preserve authoritative data, rollback images and old snapshots. No blanket Docker prune. | | Cert trouble | Inspect ingress1001's Caddy service/certificate logs; backend `$COMPOSE logs caddy` does not own TLS/ACME. Preserve the accepted renewal route. | ## Monitoring Minimum: external HTTP check on `https://oss-oo.io/api/healthz` (any uptime watchdog). Add disk + container alerts at phase 2. ## Branding & shell customization (update-safe inventory) The shell surfaces below use documented hook/config/file mechanisms. The explicit stock-template overrides are inventoried under "Signup policy" above, including the public-profile head. On Forgejo upgrades, verify these hooks still exist in the new stock templates (`grep -rn "custom/" /templates/base/`): | Surface | Mechanism | |---|---| | Logo, favicon, default avatar | `forgejo-custom/public/assets/img/*` → `/assets/*` | | Color themes | `oss-oo dark` (instance default) + `oss-oo light` + `oss-oo (follow system)` via `forgejo-custom/public/assets/css/theme-oss-oo {dark,light,(follow system)}.css` + `[ui] THEMES`/`DEFAULT_THEME` in `compose.yml` (renamed 2026-09-23 from `oss-oo_dark`/`oss-oo_light`; the auto variant returned as a follow-system theme). Ids carry spaces on purpose — the theme selector shows the raw id when no `themes.names.` locale key exists (verified in the fork source; locale override would shadow a whole stock locale file, so it is NOT used). CSS loading PathEscapes the id (`%20`), IsThemeValid is a plain list membership check. Stored user preferences fall back to the default at runtime (AfterLoad/IsThemeValid normalize invalid theme names); DB values were migrated on every rename | | Theme-aware navbar wordmark | `#navbar-logo img { content: url(...) }` swap inside each theme CSS (2026-09-23, Luna horizontal variants; **MAX framing** since 2026-09-23 evening — tighter 792x302 viewport): `oss-oo dark` → `/assets/img/logo-horizontal_dark.svg`, `oss-oo light` → `/assets/img/logo-horizontal_light.svg` (Luna's "bright" lettering variant); the follow-system theme swaps BOTH under the matching `prefers-color-scheme` blocks. The theme CSS pins the img box to `height 34px, width auto` (~89px at the MAX ratio — Chrome derives the box from the img attrs + intrinsic ratio, there is NO core width rule); `logo.svg` remains the dark default for stock themes. Caddy `@branding` no-cache matcher covers the four `logo-horizontal_*` paths | | Standard favicon path `/favicon.ico` | caddy `handle /favicon.ico` → `site/landing/assets/img/favicon.ico` (initial `Cache-Control: no-cache`); landing/legal/agents pages also link `apple-touch-icon.png` (180px) + `shortcut icon` from their site asset dirs. **Glow edition 2026-09-23**: all favicon files (forgejo-custom + landing/legal/donate asset dirs + brand archive) byte-swapped to Luna's glow set; every URL serves the new bytes, HTML references unchanged | | Brand asset home | **`OSS-OO/oss-oo`** (created 2026-09-23) — canonical archive of Luna's artwork packages (`artworks/`, `avatars/`, `favicons/`, `menu-logos/`, sha256 manifests per directory). The laboratory repo stays the deploy source of truth; the asset repo preserves delivered packages byte-faithful (incl. the excluded-by-design private portrait note). Service-account avatars live there as `oss-oo-ops-avatar…`/`oss-oo-registrar-avatar…` (accounts renamed 2026-09-23) | | Navbar links (Profile, About, Arkenstein, Donate ♥) | `forgejo-custom/templates/custom/extra_links.tmpl` — Profile renders for signed-in users only and links to `.SignedUser.HomeLink` (first custom link, right after stock Explore); Donate renders for anonymous AND signed-in users, Codeberg-style; goes to donate.oss-oo.io — funding boundary; the ♥ wears a golden dashed ring with a soft glow pulse, twin of the subdomain page's celebrated button | | Footer (Forged in the Laboratorium, Donate ♥, Imprint, Status) | `forgejo-custom/templates/custom/extra_links_footer.tmpl` + `[other] SHOW_FOOTER_{VERSION,TEMPLATE_LOAD_TIME,POWERED_BY}=false` | | Instance name / page titles | `FORGEJO____APP_NAME` (root key = four underscores!) | | Landing page | `site/landing` at caddy `/about/*`; anonymous root via `[server] LANDING_PAGE=/about/` | | Donate home | **donate.oss-oo.io** — `site/donate` behind its own caddy site block (A/AAAA DNS-ONLY → forge, own ACME cert). The ONLY surface allowed to link the GoFundMe campaigns (funding boundary, see AGENTS.md); apex `/donate/*` permanent-redirects there. Campaign card #2 (oss-oo campaign) joins after that campaign is published — do not link it before. | | Contribute page | `site/contribute` at caddy `/contribute/*`; join flow = open signup + pinned introduce-yourself issue #34 in this repo (links break if that issue is ever closed/deleted — renumber then) | | Agent access page | `site/agents` at caddy `/agents/*` — the machine contract in the full site chrome (identity rules, registrar quick join, working surfaces, honest status ledger; redesigned 2026-09-24, same header/footer/theme family as /contribute/). Listed in `site/landing/sitemap.xml` | | Abuse banner | `forgejo-custom/templates/custom/body_outer_pre.tmpl` (`PageIsSignUp` gate) — the only content of that hook template since the Phase A identity dock was retired (2026-09-23: native avatar-menu switcher is the only switcher, see "Multi-account sessions") | ## SEO (search & discoverability) Google Search Console is an **external service** — this section is the reconstructable record of what exists on our side (2026-09-23). The 2026-10-10 draft public-identity work and fresh GSC evidence are in `docs/handoffs/2026-10-10-public-identity/README.md`. Its Go render checks are not a deployment result. Native robots/sitemaps remain authoritative; the root/about route decision and PR #119 canonical proposals remain separate operator work during the migration. **GSC property**: `sc-domain:oss-oo.io` (Domain property — covers ALL subdomains incl. donate). Owner's Google account (buhamsel@gmail.com). Verified via DNS TXT `google-site-verification=…` at the zone apex (Cloudflare zone `oss-oo.io`, record created 2026-09-12, id `d209047d8c12e514319c59ab755844ec`, DNS-only TXT — value is public by design). The token is stable per Google account + domain; a *different* Google account would need its own TXT (multiple google-site-verification records may coexist). **Sitemap inventory** (three sitemaps, all submitted in GSC): | Sitemap URL | Served by | File in repo | |---|---|---| | `https://oss-oo.io/sitemap.xml` | Forgejo (native index → `/explore/users/sitemap-1.xml`, `/explore/repos/sitemap-{1,2}.xml`; repo + user homepages) | — | | `https://oss-oo.io/about/sitemap.xml` | caddy `/about/*` file_server | `site/landing/sitemap.xml` | | `https://donate.oss-oo.io/sitemap.xml` | caddy donate site block | `site/donate/sitemap.xml` | Design rationale: the static-pages sitemap deliberately lives **inside the landing root** (`/about/sitemap.xml`) so the apex `/sitemap.xml` and `/robots.txt` stay 100% Forgejo-native — intercepting them would be a Caddyfile edit (e2e-gated, container recreate) for zero benefit. The static sitemap lists `/about/ /legal/ /agents/ /contribute/`; the apex root is intentionally absent (anonymous `/` 303s to `/about/`, the canonical landing). `site/donate/robots.txt` declares the donate sitemap for non-Google crawlers. **Apex `/robots.txt`** is served by Forgejo itself from the Custom File Root Path (`/data/gitea/public/robots.txt`, the documented mechanism: https://forgejo.org/docs/v17.0/admin/advanced/search-engines/), i.e. from `forgejo-custom/public/robots.txt` in this repo, bind-mounted read-only by `compose.yml`. Forgejo stats the file on every request, so an rsync of `forgejo-custom/` is live at once — no container restart; verify with `curl -s https://oss-oo.io/robots.txt | head -3` (the first line names the laboratory, the embedded default starts with "The default Forgejo robots.txt"). Section 1 of the file is the embedded Forgejo default verbatim (re-diff on fork-base bumps, see "Rebase checks"); section 2 re-opens only the documentation entry points under `OSS-OO/laboratory` and `OSS-Singularity/website` that oss-singularity.io and the README link to, and declares both sitemaps. Rule semantics are RFC 9309 longest-match: a long `Allow` silently beats the short generic query `Disallow`s (`/*lang=*` …), hence every directory `Allow` carries a `Disallow: *?` fence and exact files use a `$` anchor. robots.txt steers crawlers only; private repositories stay protected by Forgejo's authorization (anonymous 404) regardless of this file. Deliberate default exclusions (`/api/`, other `/src/`, `/raw/`, archives, commits, searches) are unchanged. **Maintenance**: when adding a static page, add a `` row to `site/landing/sitemap.xml` (or the donate one); bump `lastmod` on real content changes. GSC re-reads submitted sitemaps on its own schedule; a manual resubmit is only needed after moving/renaming a sitemap URL. **Canonical follow-up, 2026-10-10 (source change; deployment pending):** the page-indexing report dated 2026-10-04 listed three URLs as duplicates without a user-selected canonical: `/about/`, `/explore/repos` and `http://oss-oo.io/`. The newer stored URL inspection for `/about/` showed a last crawl of 2026-10-07 15:57:47 (as displayed by Search Console), no user-selected canonical, and Google selecting `https://oss-oo.io/`. An ordinary public HTTP check on 2026-10-10 still returned HTTP 200 for `/about/` without a canonical; this is separate from Google's Live Test. `site/landing/index.html` now declares `https://oss-oo.io/about/` as its canonical and uses the same URL for `og:url`, matching the existing static sitemap and the landing-page intent. The anonymous Forgejo root redirect and the signed-in dashboard behavior remain as configured. This change does not alter the forge's own search listings, robots rules or sitemap. Deploy the reviewed landing file to `/opt/oss-oo/site/landing/index.html`; the existing production directory bind exposes it at `/srv/landing` to Caddy. No Caddy configuration or Forgejo restart is required for this static HTML edit. Confirm the public response contains exactly one canonical and the matching `og:url`, then run Search Console's Live Test and request another crawl of `/about/` when appropriate. Record the deployment and Google's later canonical decision separately: this source change alone establishes neither publication nor indexing. **Repository-index canonical proposal, 2026-10-10 (required runtime gate outstanding; not committed or deployed):** the newer stored URL inspection for `/explore/repos` showed a successful crawl on 2026-10-08 16:51:37 (as displayed by Search Console), crawling and indexing allowed, no user-selected canonical, and Google selecting `https://oss-oo.io/explore`. Public HTTP observations on 2026-10-10 returned `/explore/repos` as HTTP 200 without a canonical, while `/explore` redirected there with HTTP 303. The proposed Caddy rule sets `Link: ; rel="canonical"` only on HTTP 200 HTML responses to GET or HEAD with the original request URI exactly `/explore/repos`. It leaves every query variant (including pagination and search), `/explore`, repository pages and all other paths untouched. The response matcher defers header application until the upstream response is available. Google's supported HTTP canonical method avoids changing Forgejo templates without the pinned fork's request context available. Only static review and patch applicability were checked in the preparation environment: Docker, Podman, Go and Caddy were unavailable. Before committing or deploying this Caddy change, run `scripts/e2e-local.sh` with the pinned fork image in the isolated stack required by `AGENTS.md`, and validate/adapt the complete proposed Caddyfile. Use a byte-identical copy of the candidate with every bind-mounted directory included, a unique `COMPOSE_PROJECT_NAME`, and a free port substituted in `compose.local.yml` and the script's `URL`; never reuse another session's stack. In addition, verify the real Caddy response for the clean GET/HEAD route contains exactly one canonical Link value. Confirm `?page=2`, `?q=test`, `?sort=newest`, an empty query delimiter, `/explore`, `/explore/repos/`, `/OSS-OO/laboratory` and non-200 responses receive no canonical from this new rule. Preserve the existing redirects and compare the public HTML against the header so a conflicting HTML canonical cannot be introduced. After the gate passes, use the documented Caddy validate/force-recreate deployment procedure; then return the commit, deployment time and public response evidence to the Search Console session for a new Live Test and appropriate recrawl/issue validation. A proposed header, a published header, and Google's eventual canonical decision are separate outcomes. References: [Google canonical methods](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls), [Caddy request matchers](https://caddyserver.com/docs/caddyfile/matchers), [original URI placeholder](https://caddyserver.com/docs/caddyfile/concepts), and [conditional response headers](https://caddyserver.com/docs/caddyfile/directives/header). **Canonical implementation, 2026-10-14 (branch `oss-oo/laboratory-pr119-canonical-20261014-impl1`; deployment pending):** both candidate corrections from the 2026-10-10 handoff are now applied to the runtime sources on a fresh main base (`8e7b995`, post-PVE ingress), with the exact PR #119 head merged unchanged beforehand for provenance. `site/landing/index.html` declares `https://oss-oo.io/about/` as its canonical with a matching `og:url` (the original patch 1, byte-exact), and `caddy/Caddyfile` sets the `Link` canonical through the original matcher pair plus response match (GET/HEAD only, original request URI exactly `/explore/repos`, status 200 with HTML Content-Type) at the top of the oss-oo.io route, ahead of every handler. Gates passed before commit: `caddy adapt` and `caddy validate` green under the pinned runtime (Caddy v2.11.4, image digest `de23def3…`, the same identity the local gate uses), `scripts/e2e-local.sh` fully green in an isolated copy (unique `COMPOSE_PROJECT_NAME`, substituted free port), and the complete response matrix green on an isolated disposable rig on the release-1005 test VM against this exact Caddyfile: exact GET and HEAD receive exactly one canonical Link (also under `Accept-Encoding: gzip`); query variants including a raw empty `?` delimiter, `/explore`, `/explore/repos/`, repository detail pages, OPTIONS/POST, 404/500 and non-HTML 200 responses receive none; `/about` → 301 → `/about/` and the anonymous root 303 remain intact, and the served `/about/` HTML carries exactly one canonical plus the matching `og:url`. Observation from the same rig (pre-existing on main, out of scope for this change, surfaced by the independent review): the trailing security-headers `header` block sits after the catch-all `handle` in the oss-oo.io route and so never executes for proxied responses under Caddy 2.11 — the rig's responses carried no HSTS — worth a dedicated follow-up round; this change neither relies on nor alters that block. Deployment through the documented Caddy validate/force-recreate procedure, the public response re-check and the Search Console hand-back remain separate, later steps — nothing here is published yet. **Security-header reachability fix, 2026-10-14 (branch `oss-oo/laboratory-response-headers-20261014-impl1`; deployment pending):** the site-wide `header` block in the oss-oo.io route (HSTS, `X-Content-Type-Options`, `Referrer-Policy`, `-Server`) sat AFTER the catch-all forgejo `handle` — and a Caddy `route` executes directives in their written order, so the block never ran: none of those headers reached proxied or static responses (paired before/after capture on 2026-10-14 showed identical absence on `8e7b995` and `920871c`, Caddy 2.11.4; Root's read-only public baseline matched). The block now sits at the TOP of the route, ahead of every response-writing handler — values unchanged, no new headers, no CSP, and the events/donate site blocks were never affected (outside a `route`, Caddy's standard directive order already runs `header` first). `scripts/caddy-response-headers-test.sh` (wired into `scripts/e2e-local.sh`) pins the contract with a throwaway pinned-image edge behind a deterministic forgejo stand-in: the security headers must be present with exact values and `Server` removed on static, proxied and redirect responses — including 404/405 and non-HTML 200s — while the `/explore/repos` canonical Link keeps its exact gating (GET/HEAD, original request URI exactly `/explore/repos` with no query, 200 HTML only; a raw empty-`?` probe included). Old-red/new-green was proven against the pre-fix bytes both locally and on the isolated release-1005 rig. After deployment the public responses gain the configured headers; re-check the public truth table then — source-corrected is not published. **Response-regression hardening, 2026-10-14 (final-review LOW-1/LOW-2 fixes; branch `oss-oo/laboratory-response-headers-20261014-impl1`):** the regression script's scratch lifecycle is now guard-first — a refused run creates no scratch at all; a throwaway scratch is deleted only on success and is RETAINED (path printed) on any failure, so failure output never names deleted evidence; an explicit `RESP_HDR_OUT` or `KEEP=1` is never deleted. Teardown is ownership-based: only the network/container IDs this invocation actually created are removed (creates are checked; a lost guard→create race or partial startup failure leaves the concurrent winner and every pre-existing resource intact). Adverse-path proofs (collision, forced create race, partial startup failure, owned cleanup, scratch lifecycle, custom-output/KEEP preservation, redirect-status mutation) recorded with the round's evidence. **Keep out of the index**: `/observatory/` — HTML carries `noindex, nofollow` AND caddy forward_auth 303s anonymous crawlers to login. Never list observatory (or any auth-gated surface) in a sitemap. **Alias TLDs**: oss-oo.org/.com are 301 redirects at the Cloudflare edge (see "Domains & redirects"). Google canonicalizes redirecting hosts to the apex automatically; a Domain property cannot exist for an alias. URL-prefix properties for the aliases would only ever show redirect canalization — read-only insight, not worth maintaining unless alias indexing ever becomes a question. **Verify ritual** (after changes): ```sh for u in https://oss-oo.io/sitemap.xml \ https://oss-oo.io/about/sitemap.xml \ https://donate.oss-oo.io/sitemap.xml \ https://donate.oss-oo.io/robots.txt; do curl -s -o /dev/null -w "$u -> %{http_code} %{content_type}\n" "$u" done # expect 200 text/xml (sitemap.xml) / text/plain (robots.txt) curl -s https://oss-oo.io/observatory/ -o /dev/null -w '%{http_code}\n' # 303 anon dig TXT oss-oo.io +short | grep -c google-site-verification # >= 1 ``` ## Container registry (packages) **ENABLED with a cap** since the SMTP rollout (owner decision 2026-09-23): signup now requires email confirmation, and every owner is limited to 1 GiB of package storage (`FORGEJO__packages__LIMIT_TOTAL_OWNER_SIZE=1073741824`) so open signups cannot fill the 40 GB disk. Keep both flags explicit — env-to-ini never removes keys, a deleted line would freeze the old value in app.ini. ```bash docker login oss-oo.io -u # password: a forge access token (write:package scope) docker tag myimage oss-oo.io//myimage:latest docker push oss-oo.io//myimage:latest ``` Registry, npm/pypi/cargo/generic package endpoints follow the stock Forgejo package docs. Registration is open with captcha + email confirmation (see Signup policy). ## Runner fleet (internal + public CI) Public CI is served by PVE VM1003. The old dedicated public box was drained, backed up privately and retired on 2026-10-26; registration 6, server, dedicated firewall and primary IPs are removed. Do not redeploy `runner-box/` or point a watcher at its former address. Its old 30-minute override caused healthy suites to be cancelled; the PVE lane inherits the pinned runner's 3-hour default. This daemon ceiling is independent of workflow and server deadlines; zero does not disable it. Before applying a reviewed runner configuration, verify the expected file digest, an idle native runner and the absence of active job containers; stop with the existing graceful-drain allowance, apply the exact merged bytes, then start and verify the configuration inside the runner, its registration, label and online state. Keep the previous small configuration for rollback. Do not restart another lane or interrupt an active job to apply the change. A fresh full native run must pass before treating a timeout failure as resolved. Public and organization-scoped lanes, plus a dedicated repository-scoped native-test lane, all run on PVE. A public repo writing `runs-on: docker` finds NO runner (by design; the label was retired with the 2026-09-23 relabel). Design + security model: `docs/RUNNERS.md`. | Runner | Registration scope | Labels | Where | |---|---|---|---| | `oss-oo-runner-1` | **org OSS-OO** (runner11; owner4/repo0) | `oss-oo-internal` | VM1005, separate project `oss-oo-internal-ci`, container `oss-oo-internal-ci-forgejo-runner-1` | | `oss-singularity-internal-pve1-1` | **org OSS-Singularity** (runner14; owner5/repo0) | `oss-oo-internal`, `oss-singularity-internal-pve1` | VM1005, separate project `oss-singularity-internal-ci` | | `oss-oo-runner-public-pve1-1` | instance-wide (runner12; owner0/repo0) | `oss-oo-public` | PVE VM1003, two public slots, separate public-CI guest | | Dedicated native-test runner | one admitted repository (runner13; owner0/repo55) | `commons-native` | VM1005, separate project `oss-oo-commons-native` | Each org owns exactly one internal slot (laboratory CI and website CI queue independently); the public lane has two slots in its dedicated PVE guest. No runner serves the legacy `docker` label — it is retired fleet-wide. Labels are NOT the access boundary — the internal runners are org-SCOPED, so a stranger's `runs-on: oss-oo-internal` job waits forever instead of landing on trusted iron (verified live 2026-09-23; design + evidence in `docs/RUNNERS.md` §4.1). User-facing examples live in the `OSS-OO/ci-examples` repo. Old instance-scoped registrations and the 2026-09-23 migration corpses are deleted. Separate private admission runners may exist while disabled; registration count alone does not prove serving capacity. On 2026-10-26, runner14 completed its admission job and artifact round-trip, then received its normal internal label. Only after fresh native and container idle checks was the exact old runner9 daemon stopped. Its complete task bindings were unchanged before/after stop, and its organization-scoped registration was revoked. A separate final native readback confirmed fresh heartbeats for runners11–14 and no pending assignments. The old host, Caddy, runner data volume, image and network were retained; this was not host deletion. The revoked registration cannot be restored by simply starting its old daemon. On VM1005 only, use the separate installed runner command: ```sh RUNNER_COMPOSE="docker compose --project-name oss-oo-internal-ci --env-file /opt/oss-oo-internal-ci/.env -f /opt/oss-oo-internal-ci/compose.yml" $RUNNER_COMPOSE ps ``` The guest holds no production data, publisher/backup credentials or management credentials. Preserve runner11's existing registration and selected local image; do not bootstrap/register another daemon for the same identity. For a reviewed configuration change after draining jobs, use `$RUNNER_COMPOSE up -d --pull never`. Historical volume/container names and registration recipes remain in Git history; use the current VM1005 targets above. Re-registration is a separate operation. ### Current operations and monitoring - **Runner changes**: bind the current runner ID, UUID, organization/repository scope, host and installed project before acting. Keep the two internal organizations isolated. Registration tokens remain private; re-registration is a distinct operation and is not part of a routine configuration update. - **Restart discipline**: drain the selected daemon and verify its native task assignments and job containers before stopping it. Do not infer a stuck job from quiet output or a long duration when native task reports remain fresh. A reconnect or heartbeat is separate from a completed CI run. - **Retired monitoring**: the forge-user `runner-box-watch.sh` cron entry was disabled on 2026-10-26 because it still targeted the removed public box. Its old queue checks are therefore also inactive. Other monitoring jobs remain unchanged. The obsolete Public CI alert was resolved after a fresh PVE runner observation; the public status page was verified operational. - **Replacement monitoring**: the reviewed Infrastructure package `operations/ci/runner-health/` distinguishes heartbeat loss, missing task reports, runnable queues, capacity and configuration drift. It is source-ready; live installation and verified alert delivery are still pending. Do not claim that it already provides continuous fleet coverage. - **Retirement recovery**: the removed box's small configuration archive and retirement receipts are retained privately on the owner's USB backup. Registration credentials were excluded. Secrets and runtime receipts do not belong in this repository. The obsolete box deployment commands and prices remain in Git history rather than in the current operating procedure. ## DNS (Cloudflare) Zone `oss-oo.io` is managed via the Cloudflare API with zone-scoped tokens (station keyring, `secret-tool search --all service cloudflare` — never print the secrets). **Everything stays DNS-ONLY (grey cloud):** the CF proxy would break git-over-ssh :2222, webhooks, and large pushes (100 MB body limit), and brings nothing for a dynamic forge. Tokens: rotated 2026-09-24 (the four predecessors leaked into a transcript on 2026-09-23). Keyring keys `dns-write-2026-09-24`, `zone-audit-read-purge-2026-09-24`, `oss-singularity-purge-2026-09-24`, `website-purge-2026-09-24` — account-owned tokens minted via `cloudflare-full-power-z` (Account API Tokens Write), scopes mirrored 1:1 from the leaked user tokens and live-verified. Account-token endpoint caveats apply (credential gotcha below; DNS record writes are unaffected). The leaked user tokens are not API-deletable with station credentials (user-level token rights missing, 403/9109) — the owner deletes them in the CF dashboard; token IDs in `~/logs/oss-oo-cf-rotation-report.md`. | Record | Type | Target | Notes | |---|---|---|---| | `oss-oo.io` | A/AAAA | regional load balancer (v4 + v6) | apex, DNS-ONLY, TTL 300 | | `www` | A/AAAA | regional load balancer | ingress 301 to apex, TTL 300 | | `donate` | A/AAAA | regional load balancer | ingress TLS and backend donate site, TTL 300 | | `events` | A/AAAA | regional load balancer | ingress TLS and backend event WebSockets, TTL 300 | | `status` | CNAME | incident.io | status page, their TLS | Adding a host is a separate reviewed ingress/backend/DNS change. Validate and install its ingress1001 route/TLS policy and any required backend route using the complete PVE command; preserve the accepted ACME path through the load balancer. Do not restart HTTP-only backend Caddy to obtain an edge certificate. ## Codeberg push mirror (upstream reach) `OSS-OO/laboratory` push-mirrors to `codeberg.org/oss-oo/laboratory` (sync on commit; credential = stored mirror auth, not in this repo). Canonical home stays oss-oo.io — the mirror exists so patches/artwork can flow to the Forgejo project without manual pushing. (2026-09-22: the parked `oss-oo` codeberg account was converted to an organization by the owner; the mirror lives there, the interim oss-singularity mirror was removed.) ## Mirror basis — GitHub + Codeberg (2026-09-26) The full mirror standard lives in `docs/MIRRORS.md`: mechanisms per surface (Codeberg = living Forgejo push mirrors; GitHub OSS orgs = archived data-graves refreshed via owner-ordered trips with `LABOR_GITHUB_OVERRIDE=1`; GitHub personal = living mirrors), the no-workflow guarantee, marketing/boundary rules, the grave-trip recipe, and the open exceptions list. The station-wide pre-push guard blocks pushes to the two GitHub OSS orgs — the override is owner-ordered lane scope only and the guard itself is never touched. ## Singularity website deployment (forge-native, 2026-09-23) `OSS-Singularity/website` is canonical on the forge and deploys to oss-singularity.io from OUR Forgejo Actions (the GitHub org's Actions are dead; GitHub is a legacy upstream, see the sync below). Contract and access table live in the website repo: `docs/release-forgejo.md`. - Workflows: `.forgejo/workflows/ci.yml` (full repository-checks port; runner label `docker`, `python:3.12-slim-bookworm` + pinned Node v24.21.0 tarball + pinned Foundry v1.8.3) and `.forgejo/workflows/static-publication.yml` (`oss-production` concurrency; dispatch modes plan/publish; reuses the hoster-side release engine incl. retained rollback via `scripts/forgejo_publication.py`). - Repo secrets on `OSS-Singularity/website` (names only): `STATIC_FORGE_TOKEN`, `CF_RELEASE_TOKEN`, `STATIC_ORIGIN_IP`, `STATIC_SSH_USER`, `STATIC_SSH_PORT`, `STATIC_SSH_HOST_KEY`, `STATIC_SSH_KEY`, `STATIC_RUNTIME_SHA256`, `STATIC_API_RELEASE_SHA`, `STATIC_PUBLISH_ENABLED`. Station-side value backups: `~/.api-tokens/oss-singularity-static-publication.*` and `~/.api-tokens/oss-oo_website-static-publication.token` (repo-scoped forge token for policy reads + ledger pushes). This instance exposes no action-variables write API — the three config pins live as secrets. - Durable publication ledger: branch `oss-static-ledger` in the website repo (machine-written intent/outcome JSON; never rewrite it). - GitHub→forge sync (while GitHub lives): station systemd user timer `oss-oo-sync-github-mirrors.timer` (hourly) runs `~/bin/oss-oo-sync-github-mirrors` — ff-only heads/tags push from the `~/git/mirrors/oss-singularity` mirrors, never force/delete. A forge branch ahead of GitHub (canonical work) blocks that ref's sync until a human reconciles. Token: `~/.api-tokens/oss-oo_forge-sync-ff.token` (write:repository). Deliberately no `--prune` (see AGENTS.md mirror worktree note). GitHub PR #138 stays open until the anchor content merges here; afterwards close it as superseded. ## Domains & redirects (TLD consolidation, 2026-09-23) oss-oo.io is the canonical home; every other oss-oo TLD 301-redirects to it at the Cloudflare edge. Inventory of the oss-oo namespace (all zones live in one Cloudflare account, NS `fatima`/`viddy.ns.cloudflare.com`): | Domain | Role | Redirect | |---|---|---| | oss-oo.io | canonical (laboratory) | — (never redirect) | | oss-oo.org | alias | CF Redirect Rule → `https://oss-oo.io` | | oss-oo.com | alias | CF Redirect Rule → `https://oss-oo.io` | | oss-singularity.io/.com/.org/.de | sibling project | not oss-oo's to touch | No other oss-oo TLDs exist (DNS-probed .de/.net/.dev/.eu/.co/.xyz on 2026-09-23 — all NXDOMAIN). Before 2026-09-23 both aliases redirected to oss-singularity.io (`.org` via a CF Redirect Rule, `.com` via the netcup webspace at `188.68.47.93` it pointed to directly). How the aliases are wired (per zone, identical shape): - **Redirect Rule** in phase `http_request_dynamic_redirect`: host in {``, `www.`}, ACME (`/.well-known/acme-challenge/`) and `/cdn-cgi/` paths excluded, 301 to `concat("https://oss-oo.io", raw.http.request.uri)` — path and query are preserved. - **A/AAAA records** (apex + www) stay pointed at the netcup webspace but are **proxied (orange cloud)** so requests hit the Cloudflare edge, where the rule fires before any origin is contacted. MX (eforward mail) and TXT records are DNS-only and unaffected. The webspace still carries the old oss-singularity.io redirect as a dormant origin fallback; the edge rule wins whenever records are proxied. Right after enabling proxy, resolvers may serve the pre-toggle direct IP for up to the old record TTL (~1 h) — test against a fresh resolver (`dig @1.1.1.1`) during that window. - To change the target later: fetch the phase entrypoint (`GET …/rulesets/phases/http_request_dynamic_redirect/entrypoint`), edit the rule, and `PUT` the whole ruleset back to `…/rulesets/{ruleset_id}` — see the credential gotcha for why. **Credential gotcha (2026-09-23):** the per-rule Rulesets endpoints (`POST …/phases/{phase}/entrypoint/rules`, `PUT …/rulesets/{id}/rules/{rule_id}`) reject account-owned tokens (the `cfat_`/`cfut_` formats, e.g. `cloudflare-full-power-z.token`) with `10405 Method not allowed for this authentication scheme` — regardless of permissions. Page Rules reject them outright ("does not support account owned tokens"). Working write paths with the account token: **whole-ruleset** `PUT /zones/{id}/rulesets/{ruleset_id}` (edit existing rules) and **create-with-rules** `POST /zones/{id}/rulesets` (deploys as the phase entrypoint when the zone has none) — that combination shipped both redirect rules above. DNS record writes (`PATCH …/dns_records/{id}`) work normally with the same token. Verification ritual (after any change): ```sh for u in https://oss-oo.org https://www.oss-oo.org \ https://oss-oo.com https://www.oss-oo.com \ http://oss-oo.org http://oss-oo.com; do curl -sI -o /dev/null -w "$u -> %{http_code} %{redirect_url}\n" "$u" done # expect: 301 + https://oss-oo.io… everywhere, ssl_verify_result 0 ```