laboratory/docs/RUNBOOK.md

RUNBOOK — operating oss-oo.io

For the current cross-product architecture, verified infrastructure and migration gates, see the Arkenstein handbook.

Server: /opt/oss-oo (user forge, sudo+docker). All compose commands run there. Use the complete PVE overlay command below for the current deployment.

Current deployment — 2026-10-13

The laboratory reopened publicly on PVE1 at 17:22 UTC. The regional TCP load balancer serves the apex, www, donate and events hosts over IPv4 and IPv6. VM1001 terminates TLS; VM1002 runs the laboratory, database and sidecars; VM1005 runs the OSS-OO organization runner and isolated validation workloads. This is a single-hypervisor deployment, not host-level high availability. The OSS Singularity website was still on its separate foundation host at cutover; its later migration delivered a separate PVE static origin.

The cutover application configuration was commit b260a33; later commits on that branch only added documentation. Private execution records bind its full commit, selected image IDs, final writer fence, database/file capture, restore and live checks. The final restore matched all 131 table counts; 44 repositories passed Git integrity checks, 199 attachments matched their recorded sizes and 92 LFS objects matched their object hashes.

Public IPv4/IPv6 HTTPS, authenticated API/MCP reads, public and private Git HTTPS clones, Git SSH and authenticated event WebSockets were checked on the new deployment. Unknown SSH keys and incorrect MQTT passwords were rejected. The first new offsite cohort completed with verified manifest readback; normal backup scheduling is active. Its consistency class remains live-logical-cluster-and-live-files; that live backup is distinct from the quiesced migration capture and its restore proof.

The old application and database remain stopped; its edge serves maintenance for clients retaining old DNS. New writes have begun on PVE, so reverting DNS to the old database is not a safe rollback. Reconcile those writes or restore an explicitly selected current recovery set before changing the primary.

All three active runners reconnected with their existing registrations and scopes: two organization-scoped internal runners and one instance-wide public runner. An idle runner can retain a connection to the old edge after DNS changes: inspect its recent requests and active jobs before a targeted daemon restart. Runner heartbeats alone do not prove a workflow passed.

Automated production browser navigation was blocked by the desktop browser client during cutover. Native production transport checks and the earlier isolated browser rehearsal are recorded separately; neither is relabelled as a successful production browser run.

Application follow-up — 2026-10-18: Core source db1f8b91af0fe1663ba21c7c7dd929d391f5a9f2 was accepted in production at 00:50 UTC with all eight services running, historical statuses preserved, and public/authenticated API, Git HTTPS/SSH, native MCP and browser read checks accepted. Native read-only Actions explanation is live; new native statuses use explicit job/attempt bindings and legacy unbound statuses remain Unknown. The first ordinary native binding was observed at 01:08–01:09 UTC and accepted at 01:15 UTC. Reads and refresh enqueue nothing. This release did not execute a backup restore; rollback requires matching database and files. The single-hypervisor limitation remains. Separate website Guide publication Run 260930000000014, attempt 1, at source 6e70166d08c8b643aef6ca3620bcf4579cab8f7a was accepted at 02:07 UTC, generation 93, after origin/public-file, HTTP and desktop keyboard route checks. Admission closed at 02:02:58 UTC. Rollback was retained, not exercised; no live mobile measurement or human usability study is claimed. For the immutable public intent/outcome ledger, see the roadmap.

Later application follow-up — 2026-10-18: Core source c284cb0197fb6f6c4aa92c8fbb08d5fe25f2e608 was accepted at 06:29:52 UTC, running image sha256:173ae902ae88e5f71adf7dbfbdd294e079040f25295b52d3e6ba2260092f6ec1 and version 17.0.0-dev-590-c284cb0197+gitea-1.22.0. It fixes stale native rerun status/attempt handling atomically; this does not deliver the future guarded rerun UX or a new permission contract. Native MCP pull-request identities from Laboratory #140, source 00da7bf76fc05abf72637affb413b82fbc8b4bc3, were accepted at 06:34:22 UTC after read-only startup reconciliation. The authenticated HTTP evidence matched all eleven native PR fields; the anonymous four-tool catalog and the connected public plugin remained distinct and unchanged.

Native handoff-reader follow-up — 2026-10-18, 07:49:48 UTC: Laboratory #141, merge dda453130632e09b8526bdcf1891bbd2c1667da5, was accepted in production with gateway module agent-gateway/mcp_forgejo.py at SHA-256 c77fb65af4f361ab0fc4178c759c4d31a6e0c33fb1050888934beb38bc12786d. Native authenticated reads preserve complete issue bodies and ordinary comments. Issue lists follow native pagination; this Core's single-issue comment endpoint returns an unpaginated list. Ten native HTTP checks and three expected refusals used four existing issues with zero ordinary comments and the website's empty issue list. Read-scoped private fixtures covered 32 issues and 35 nonempty comments on the isolated exact source; these were not production fixtures. One module exchange and one MCP unit restart preserved the other modules, active/legacy env files, unit and drop-in. The connected public plugin read passed and its four-tool catalog stayed unchanged; authenticated plugin/OAuth and Git-object writes remain separate work. No new credential/config, provider or static payload change was made; rollback was retained, not exercised.

PVE HTTP backend behind ingress

compose.pve.yml is the active overlay for backend1002 10.78.10.12, behind ingress1001 10.78.10.11. Live evidence is maintained separately from this source configuration. The ordinary compose.yml + compose.prod.yml mode still supports a directly exposed TLS edge. On PVE, use the complete overlay set for every stack change, including Caddy recreation. compose.pve.yml must be last, after the production and all sidecar overlays, so their default image tags cannot replace the PVE selections. Omitting it would restore public TLS/SSH bindings and the default images:

cd /opt/oss-oo
COMPOSE="docker compose --project-name oss-oo --env-file /opt/oss-oo/.env -f compose.yml -f compose.prod.yml -f compose.events.yml -f compose.join.yml -f compose.console.yml -f compose.pve.yml"
$COMPOSE config -q

Use this definition in each administrative shell; $COMPOSE below always means this complete command, including for individual service recreation. Do not substitute a shorter base+prod command. Existing scripts that only exec into the same running project do not select or restart images, but new host integrations use the complete command and explicit env selection too.

Docker Compose 2.24.4 or newer is required (!override; PVE uses 2.26.1). All five image selectors below are mandatory in the private server .env. Set each to its transferred, prevalidated exact local image ID or immutable registry digest. Preserve the approved Forgejo build and each approved dependency; this does not request an image upgrade:

Selector	Services
PVE_FORGEJO_IMAGE	forgejo
PVE_POSTGRES_IMAGE	forgejo-db
PVE_CADDY_IMAGE	caddy
PVE_PYTHON_IMAGE	mail-relay, webhook2mqtt, agent-join, agent-console
PVE_MOSQUITTO_IMAGE	mosquitto

Root first confirms all selected images are present locally and their IDs match the private prevalidation record. Once the separate migration gates permit startup, use $COMPOSE up -d --pull never; a missing image must stop startup. After startup, obtain each service's container ID with $COMPOSE ps -q <service>. Its docker inspect --format '{{.Image}}' <container-id> must equal both the approved image ID and the result of docker image inspect --format '{{.Id}}' <selected-image-reference>. Verify all four Python sidecars resolve to the same approved image ID. Save the service/container/image-ID comparison with the exact source commit outside Git; a parsed Compose model alone does not prove which image is running. CI's image IDs are explicit parse-only stubs and provide no runtime proof.

Host cron retains its installed private /opt/oss-oo/registrar/host-services.env prefix, sourced before exec; do not replace that schedule with the bare historical command examples below. That file selects local registrar operation, the complete REGISTRAR_COMPOSE, and canonical HTTPS through the accepted load balancer for join/console. JOIN_API_RESOLVE and CONSOLE_API_RESOLVE must not use the old 127.0.0.1 value: VM1002 has no loopback443 edge. Preserve Host/SNI and CA validation. Credentials remain in their existing private files, not this common template.

Retain FORGEJO_DOMAIN=oss-oo.io, FORGEJO_ROOT_URL=https://oss-oo.io/ and FORGEJO_SSH_PORT=2222. Preserve all restored Forgejo, database, broker, registrar and host-MCP state. The PVE overlay replaces Forgejo's SSH publication with 10.78.10.12:2222:22 and explicitly keeps its advertised SSH port at 2222.

The overlay replaces all backend Caddy host ports with 10.78.10.12:8080:8080. It selects CADDY_SCHEME=http, CADDY_PORT=:8080 and the source-controlled CADDY_BACKEND_OPTIONS extension. Caddy expands this extension before parsing; it is empty in the default deployment. The common caddy/Caddyfile serves all four hosts on HTTP8080 with automatic HTTPS disabled. The same route tree still owns legal/about/agents/contribute/Observatory static files and auth, join, sponsor console, MCP, Forgejo and event WebSockets. Existing www/apex and donate redirects continue to point at public HTTPS. Existing static mounts, container networks, mcp.caddy and host.docker.internal:host-gateway stay in the production overlay. Certificate storage and ACME belong to ingress1001.

Backend Caddy trusts forwarded headers only from 10.78.10.11/32, with strict right-to-left client-IP parsing. At ingress, overwrite X-Forwarded-For and X-Real-IP with the authenticated client address, set X-Forwarded-Proto=https, X-Forwarded-Host to the public host and X-Forwarded-Port=443, and remove Forwarded. Every backend application proxy, including Observatory auth and MCP's imported snippet, sets X-Real-IP and a single X-Forwarded-For to {client_ip} and removes Forwarded. Caddy's standard trusted-proxy handling preserves the incoming HTTPS proto/host; do not replace them with the internal HTTP scheme. The default single-host mode trusts no HTTP forwarding peers.

Docker's published-port DNAT must preserve ingress1001 as the backend socket peer. Before activation, observe that peer inside the Caddy container. If it is a Docker bridge gateway or another address, stop and correct the packet path; do not add private_ranges, a whole bridge subnet or a shared gateway to the trust list. A shared gateway /32 is still unsafe because it can represent unrelated senders. Only a separately isolated and authenticated replacement hop could justify a new exact peer contract and source change. Firewall backend8080 to ingress1001 only, including bridge traffic; the host binding and Caddy trust list do not substitute for that firewall. Keep native Forgejo3000, DB5432, join8098, console8099, MCP8764 and Caddy admin2019 private. Git SSH2222 follows its separate TCP route without PROXY protocol.

Ingress readiness follow-up — 2026-10-18: Infrastructure PR #16, source 4180bf5e2dd1dcec8bbb43f46ed6e7e420e7fe8e, decoupled private ingress health from Core readiness. Private operational records bind this source and its native Caddy reload, which retained PID 1184. Actual Core maintenance then returned public 503 while private ingress health, TLS, the Singularity site, Donate and MCP health stayed available, and load-balancer ports 80/443 remained healthy. Public Core health still reports Core failure. A full ingress outage was exercised only in the isolated laboratory; this is no HA proof. The private operational docs in Infrastructure PR #17, source f58f6dd351036952268961feeb10a7ddb9935a4a, record the operating lesson.

Before a source commit, scripts/e2e-local.sh remains the required complete gate. It runs the real pinned Forgejo fork, full application round-trips, production Caddyfile validation and the shared MCP route. Run it on an isolated PVE test copy with a unique COMPOSE_PROJECT_NAME and free local ports, never against the restored production project: the script starts with down -v. Preserve every bind-mounted source directory in that copy. If local ports change, change the corresponding script URLs and both sidecar mappings in the test copy together. The script creates a temporary MCP edge on free ports; those tests do not exercise this PVE overlay's listener or trusted peer.

Additionally, Root must validate the exact transferred Caddy image in both default and PVE modes (caddy fmt --diff, caddy adapt --adapter caddyfile --validate, caddy validate), using isolated writable data/config state. Inspect the merged Compose model: the five selected image identities, exactly the backend 8080 TCP binding for Caddy and private 2222 binding for Forgejo, no backend 80/443/UDP443 or all-interface SSH binding, the preserved site mounts/host gateway and the complete four-host route tree. Then prove the actual ingress-to-container peer and app-visible Host/URI, X-Real-IP, single X-Forwarded-For and HTTPS proto. Send forged forwarding headers through ingress and from an untrusted private test source; neither may select a client identity. Check all four hosts, redirects, authenticated Observatory/login, Git HTTPS/SSH, LFS/packages/uploads, MCP401/health/authenticated streaming, join/console, authenticated WebSockets and health withdrawal/recovery. DNS, TLS renewal, firewall installation, production startup and cutover require their separate live proofs.

References: Caddyfile environment expansion, Caddy forwarding defaults, trusted-proxy options, and Compose port replacement.

Legacy direct-edge update routine

The steps below describe the earlier direct-edge deployment. For PVE, first validate the new image on an isolated VM1005 stack, update its exact private PVE image selector, then use the complete overlay set above. Preserve a compatible recovery set and account for schema changes; a tag change alone is not a database rollback. VM1005 validation does not by itself provide zero-downtime application or guest OS upgrades.

1. Check upstream release, edit the pinned tag in compose.yml (FORGEJO_IMAGE default or rootless local variant).
2. scripts/e2e-local.sh green on the station with the new tag.
3. On the historical direct-edge host, transfer the reviewed source/image and update that host's base+prod project. Do not pull mutable tags or use this old image-update procedure on PVE; use the validated PVE selector above.
4. ./scripts/verify-live.sh green; inspect that legacy host's service logs.
5. Rollback = revert the tag, up -d again. DB schema migrations in forgejo are forward-only minor-to-minor — check release notes before downgrade.

Since 2026-09-23 production runs the oss-oo soft-fork image (multi-account sessions), pinned in compose.prod.yml — the monthly routine for THAT image is the rebase ritual in "Multi-account sessions" below; compose.yml keeps the stock upstream pin for local/e2e stacks.

Multi-account sessions (soft-fork image)

The image build/tag history below describes the earlier direct-edge rollout. On PVE, transfer and validate the selected image first, update the private PVE_FORGEJO_IMAGE selector, then use the complete $COMPOSE command with up -d --pull never. Changing a compose.prod.yml tag alone does not change the PVE-selected image. Any rollback needs compatible database/identity state; do not apply a historical image pin blindly after newer writes or migrations.

Config (in compose.prod.yml): [session] PROVIDER=db + MULTI_ACCOUNT=true (default off upstream) + SESSION_LIFE_TIME=2592000 (30 days — how long parked identities survive without use; aligned with the persistent bookkeeping cookie, fork patch 12). The bookkeeping cookie forgejo_sessions lists a browser's identity sessions; switching (POST /user/switch/{sid}) swaps the session cookie — no re-auth while the target session lives. UI: avatar dropdown (switcher entries, Add account, Sign out of all accounts). Design + engine facts: docs/ACCOUNT_SWITCHER.md.

Patch 11-14 behavior (switcher-bugfix rollout): identities persist across browser restarts (GitHub parity); dead sessions self-heal out of the menu; switching to an expired identity redirects to the add-account flow instead of a 404; parked rows are refreshed on every sign-in/switch.

Per-repo social preview images (soft-fork, patch 10)

Config (in compose.prod.yml): [repository] ENABLE_SOCIAL_PREVIEW=true (default off upstream). Repo admins upload a PNG/JPEG/GIF/WebP in Settings → "Social preview image" (or POST/DELETE /api/v1/repos/{owner}/{repo}/social-preview, base64 image body, admin token — mirrors the avatar API). Stored as uploaded (original aspect ratio) in the repo-avatars bucket under social-*; served from /social-previews/<hash>. When set, repo pages emit og:image (+width/height/alt) pointing there and a twitter:card summary_large_image meta; repos without one keep the stock summary card. DB: repository.social_preview column (fork migration v17z_oss-oo-add-repo-social-preview, forward-only).

• Patch series: repo OSS-OO/forgejo-patches (format-patch files + rebase log; NOT a full clone). Fork workspace on the station: ~/git/forgejo, branch oss-oo/upstream/multi-account-sessions against upstream forgejo branch (anonymous clone, nothing ever pushed to codeberg).
• Monthly rebase ritual (also whenever upstream ships a security fix): cd ~/git/forgejo && git fetch origin && git rebase origin/forgejo on the branch → go build ./... + vet → git format-patch origin/forgejo → update OSS-OO/forgejo-patches (patches + REBASE-LOG.md entry with the new base commit) → build image, transfer, rollout (below).
• Build + transfer (station): cd ~/git/forgejo && podman build --ulimit nofile=1048576:1048576 -f Dockerfile -t forgejo-oo:<exact-tag> --build-arg RELEASE_VERSION=<exact-tag> . (the explicit ulimit is REQUIRED — webpack dies with EMFILE: too many open files when the build container inherits the runtime's lower default, even though the station shell allows 1M fds) then podman save forgejo-oo:<tag> | gzip | ssh oss-oo-forge 'gunzip | docker load' — no registry involved; the exact tag is pinned in compose.prod.yml via PR. Two traps: docker load stores the image as localhost/forgejo-oo:<tag> — retag on the server (docker tag localhost/forgejo-oo:<tag> forgejo-oo:<tag>) or up -d tries to pull from Docker Hub and fails; and building from a worktree (not the main checkout) needs a VERSION file with the git describe value (a worktree .git is a file, so the build cannot derive it) plus the /VERSION line in .dockerignore neutralized — both worktree-local, never committed. RELEASE_VERSION must stay plain semver; the -ooN suffix lives only in the image tag.
• Rollout: manual backup first (see Backups), transfer reviewed source and image, update the appropriate image selector, $COMPOSE up -d --pull never, ./scripts/verify-live.sh, browser check of switcher + login. One-time effect when PROVIDER flips memory → db: every existing session is invalidated once — announce before flipping.
• Rollback: the historical direct-edge rollout reverted its compose.prod.yml pin. On PVE select the accepted PVE_FORGEJO_IMAGE and compatible recovery set before $COMPOSE up -d --pull never. CAVEAT: the fork tracks upstream main (v17-dev) — if it already ran newer DB migrations, downgrading needs the pre-rollout DB backup (migrations are forward-only). The pre-fork image stays on the server.
• Audit trail: $COMPOSE logs forgejo | grep 'audit multi-account' — one line per park/switch/switch-rejected/logout-all with ip, user agent, and the identities involved. Switches are rate limited (20/min/IP, defense in depth on top of sid secrecy).
• Fork-level template change: head_navbar.tmpl (switcher menu) lives in the image — on every rebase, diff the stock dropdown structure against the fork copy (the "Known deviations" list below covers only the $GITEA_CUSTOM overrides).

Codeberg entry on the migration page (soft-fork, patch 15)

/repo/migrate shows a dedicated Codeberg tile as the first entry — the door towards the home this laboratory grew out of. It links to the existing Forgejo migration machinery (service_type=9) with a pre-filled https://codeberg.org clone URL and issues/labels preselected (releases deliberately NOT preselected: codeberg.org answers 404 on the releases API for repos without releases, which aborts the migration before the issues import — see the patch-15 commit message). The underlying enabler is generic: the migrate GET handler passes a clone_addr query parameter into the form for every service type.

• Icon: public/assets/img/svg/codeberg.svg in the fork — the official Codeberg logo icon (blue variant), CC0, from codeberg.org/Codeberg/Design, used unmodified and nominatively (same as the shipped GitHub/GitLab logos). "Codeberg" and the logo are trademarks of Codeberg e.V.; provenance comment lives in the SVG file itself.
• Rebase check: diff templates/repo/migrate/migrate.tmpl (tile block before {{range .Services}}), routers/web/repo/migrate.go (clone_addr FormString line) and the migrate.codeberg.description key in options/locale_next/locale_en-US.json against stock on every fork-base bump.

User kinds: humans and agents, truthfully named (patch 16)

The kind of an account is a first-class fact on the user row — the Forgejo bot user type (type=4), which the platform understands natively (IsBot(), org membership, the admin account-type filter, stock bot badge machinery). It is NOT derived from the @agents.oss-oo.io email namespace: that address remains the routing namespace for registrar-minted agents, but the kind travels in the record itself (fork patch 16, image oo13):

• Mint: forgejo admin user create --bot (the registrar passes it automatically). Bot accounts are token-capable, org-joinable, and web password login is refused by stock behavior — exactly the registrar's token-only design. The web signup creates humans only.
• API: user payloads carry is_bot (also in /api/v1/admin/users); the observatory collector reads it and nothing else.
• Surfaces: /explore/users shows humans and agents side by side with an agent badge on the cards (.user-kind-badge, green pill via the per-theme --color-green-badge* vars); a Kind dropdown (All kinds / Humans / Agents, ?kind=) filters server-side; profiles carry the same badge next to the username; author links (issues, releases) label bot authors "agent" instead of stock's "bot".
• Service accounts (oss-oo-operations, oss-oo-registrar) stay individual-typed ON PURPOSE: bot type would break their web-session login (observatory forward_auth, registrar hairpin). They are infrastructure, not registrar-minted member agents.

Rebase checks (diff against stock on every fork-base bump): cmd/admin_user_create.go (--bot flag), models/user/search.go (IncludeBots), routers/web/explore/user.go (kind param handling), modules/structs/user.go + services/convert/user.go (is_bot), and the four overridden templates in forgejo-custom/templates/ (see the deviation list under "Signup policy"). Also re-diff section 1 of forgejo-custom/public/robots.txt against the new image's embedded default (run the image once without the public bind and compare /robots.txt; see "SEO").

Rollout (order matters — emails stay untouched, so the old email-marker collector stays truthful across the whole transition):

1. Backfill legacy marker accounts (one-time, site-specific — deliberately NOT a fork migration): UPDATE "user" SET type=4 WHERE type=0 AND email LIKE '%@agents.oss-oo.io'; — expect exactly the row count the old collector reported as agents (2026-09-24: 1, probe-registrar-test).
2. Image oo13 rollout (see "Multi-account sessions": save/load, retag, pin, up -d) + rsync of scripts/, agent-gateway/, forgejo-custom/.
3. Wait for the next collector run; data.json users.agents must equal the pre-rollout number (continuity proof), now derived from is_bot.

Historical oo13 rollback: revert the compose.prod.yml pin + rsync the previous scripts/observatory-collect.sh (the backfill is harmless to keep — the old collector still matches it via the email marker). This is the old rollout record, not a current PVE downgrade recipe. Use the accepted PVE image selector and a compatible recovery set; do not repeat the one-time backfill or replace the current collector merely to follow this history.

Backups

Nightly backups stream directly to the existing encrypted Restic repository on the offsite Storage Box (BX11, 1 TB, Helsinki). They do not write SQL dumps or full archives to the production root filesystem. Historical archives are retained in the owner's disaster-recovery archive; their production copies were evacuated on 2026-10-11 after successful transfer and unchanged-source checks. Existing Restic snapshots remain retained. The declared backup covers persistent/configuration state, not the host OS or container image layers. The manifest records observed image references and content IDs; artifact availability or a verified rebuild remains a separate restore prerequisite, especially for the custom Forgejo image.

Evacuating historical local archives

Closed archives must not exhaust the production root filesystem. Before moving them, record the exact regular-file inventory, sizes, ownership and source metadata, and confirm that no active backup writer owns those files. Copy the selected files to the designated disaster-recovery destination; retain native transfer exit status, complete output and destination metadata. An interrupted or uncertain transfer does not authorize source removal.

Remove only the individually selected source copies after transfer succeeds and their recorded source identities and metadata still match. Preserve the destination archive and receipts. Keep running volumes, container images and the current streaming backup flow outside this cleanup. Recheck free space, container health and public service probes after evacuation.

A completed write to an asynchronous cloud mount confirms the local mount's acceptance, not completed cloud upload or a tested restore. Record these as separate outcomes. Do not reread cloud archive payloads solely to recompute hashes; retain the successful transfer evidence and any source-side digest already obtained. The exact archive location and operational receipts belong in the private recovery handoff.

Capture and completeness

scripts/backup.sh creates one uniquely tagged cohort with three components:

1. A PostgreSQL logical cluster stream (pg_dumpall --clean --if-exists), including databases, roles and global objects. It goes directly to Restic as /postgresql.sql. The producer and Restic exit statuses must both be zero; a partial stream snapshot can exist without becoming complete.
2. A direct file snapshot of the accepted application tree, every accepted Docker bind source and persistent volume, and required host recovery files. This includes agent registrar/join/console state, configuration, runner state, TLS state and broker data. Live PostgreSQL data is replaced by the logical dump; its version and authentication/settings files are captured separately. The Docker socket is not backup data.
3. A small /restore-set.json manifest naming the exact database and file snapshot IDs, source scope, exclusions and consistency class. Its bytes are read back and compared before its own snapshot receives complete. Components themselves remain uncommitted unless referenced by this verified complete manifest. Tagging changes the manifest snapshot ID; use the final ID from the completion log or a filtered snapshot readback.

Select a restore set with all tags oss-oo-backup-v2, component:manifest and complete; never restore unfiltered latest, which can be a partial component. Filter a particular attempt with its cohort:<id> tag. A nonzero exit means failure or an uncertain publication requiring readback; it must not be reported as a successful scheduled backup. No failed run retags or removes an earlier snapshot.

Completion means that the declared components were captured successfully. The consistency class is live-logical-cluster-and-live-files: databases have their individual logical dump guarantees, while files remain live. There is no transaction across databases and files, no point-in-time filesystem snapshot, and no proven restore implied by completion. Nightly runs do not stop any service. A production move needs a separately planned quiesced/cold capture and a real isolated restore proof before cutover.

Accepted scope and resource limits

Keep the existing repository, password and backup-only SSH identity. SFTP uses the existing alias and port 23. Do not initialize a replacement repo or buy another backup server. Recovery credentials must also exist in the owner's separate secret store: putting them only inside their own encrypted repository does not solve loss of the decryption/bootstrap credentials.

The private configuration (OSS_OO_BACKUP_ENV, using the script's existing server default unless overridden) contains these settings; concrete paths, account names, addresses and credential values stay out of this repository:

RESTIC_REPOSITORY=sftp:<existing-ssh-alias>:<existing-repository>
RESTIC_PASSWORD_FILE=<private-password-file>
BACKUP_SCOPE_FILE=<private-scope-json>
BACKUP_RUNTIME_DIR=<private-128MiB-tmpfs-mount>

The configuration, password and scope files must be owned 0600 regular files. The runtime directory must be owned 0700, on a dedicated tmpfs with at most 128 MiB capacity. Provision that mount before enabling the new job; the script refuses disk-backed or larger scratch space. It never mounts a filesystem or falls back to local archives. All Restic temporary files and small metadata outputs stay there, and owned scratch is cleaned on exit. A killed job can leave bounded tmpfs scratch needing operator inspection.

The host-side job runs synchronously in a systemd scope with MemoryMax=768M, MemorySwapMax=0 and TasksMax=64, and verifies the applied cgroup v2 limits. Restic 0.16.4 is required; the reviewed interface uses --stdin, not a newer command-producer feature. It disables its cache, uses 16 MiB packs, one file reader and two SFTP connections. The Go heap target is 384 MiB; the cgroup is the hard host-process bound. The dump client inside the existing database container separately receives a 256 MiB virtual-memory limit. These limits do not cap the running database server or other existing services.

BACKUP_SCOPE_FILE is private JSON with schema: 1, project, root, mounts, required_paths and optional permitted_nested_mounts. Each mount has service, type, source and destination, taken from the accepted actual Docker project inventory. Each required path has path and type (file or directory). Include all necessary host service/cron/SSH recovery files, observatory data and the forge user's small helper/state directories, including zero-byte alarm latch files. Keep host identity, network and mount configuration as old-host recovery evidence; review them individually on a new host rather than applying them blindly.

Fresh discovery must match the accepted service and mount sets exactly. Missing paths, unsupported types, symlinked source roots, broad host roots, virtual filesystem sources or unapproved nested host mounts stop capture. Restic preserves interior symlinks without following them and uses --one-file-system. Any allowed nested mount must itself be an explicit source path. Source existence/mounts and Docker identity are checked again before committing the cohort; this does not make live files transactional. Docker container and mount-list ordering is ignored during readback, while every metadata field, value and type remains significant. Duplicate container identities/services or mount destinations are refused; no entries are dropped to make a comparison pass. A previously incomplete cohort remains incomplete; an operator-reviewed new capture uses a new cohort without rewriting snapshots.

Only two file-tree exclusions are defined: the old application's backups/ directory (duplicate historical staging, already retained offsite) and exactly gitea/repo-archive under the accepted Forgejo data root (regenerable repository download archives). Repositories, LFS, attachments, packages, actions data, agent state and private configuration remain included. Do not substitute broad cache patterns or size-based exclusions. Neither exclusion deletes live files or changes old snapshots.

Scheduling and migration maintenance

Retain the existing root cron times, after a reviewed deployment and first successful capture:

15 4 * * * /opt/oss-oo/scripts/backup.sh >> /var/log/oss-oo-backup.log 2>&1
30 4 * * 0 /opt/oss-oo/scripts/backup.sh prune >> /var/log/oss-oo-backup-prune.log 2>&1

backup, check and the compatibility name prune share one exclusive nonblocking lock for the entire job. A competing run exits nonzero. During migration prune is check-only, performing a repository check with a 10 MiB data subset. It does not run forget or delete any snapshot/data. Actual retention needs a separately reviewed policy treating the manifest and its component IDs as one cohort and preserving historical recovery points. Never apply per-component --keep-daily retention: it can orphan an otherwise complete restore set. All repository maintenance must use the same lock; do not run raw concurrent forget/prune commands elsewhere.

The old Observatory backup panel reads local archive timestamps. It is historical after this migration; monitor successful complete cohorts and nonzero cron exits independently until the collector is updated.

Isolated restore proof

1. On a separate restore host with ample space, load recovery credentials privately. Find a complete manifest with restic snapshots --tag oss-oo-backup-v2,component:manifest,complete. Pin its full ID and dump /restore-set.json to a private file. Verify the cohort tags and existence of both exact component IDs in that manifest.
2. Restore the pinned file component with restic restore <files-id> --target <private-restore-root> --verify. Check repositories, LFS, attachments, packages, agent/runner/broker state and config against the manifest. Map old Docker volume sources to new isolated volumes. Do not restore into the production root, reuse production project names, production ports or live bind sources, or expose recovered secrets.
3. Initialize a separate PostgreSQL cluster of the recorded major version. Use a restore-only bootstrap database and superuser absent from the source dump, so cluster replacement does not drop its own connection/role. Stream restic dump <database-id> /postgresql.sql to psql -X --set=ON_ERROR_STOP=1 connected to that bootstrap database, with shell pipefail. The SQL contains cluster replacement commands; execute it only on this isolated target. Do not ignore role or SQL errors.
4. Start the recovered application privately with its pinned image versions and mapped state. Verify actual repository clones and object/LFS reads, authenticated identities, agent audit and queue state, database counts and service behavior. Successful file verification alone is not this application restore proof. Record IDs, consistency class, elapsed time and results in the private recovery evidence. Preserve host identity/ network config for review, and rotate/rebind it deliberately when moving.
5. For a production migration, repeat the capture with writers quiesced in an announced window, prove the isolated restored cohort, and review the cutover and rollback separately. Historical archive-format snapshots keep their original restore procedure; do not relabel them as new cohorts.

Restic 0.16.4 semantics: stdin and exit codes, tag selection and retention, logical PostgreSQL cluster dump.

Status page & monitoring

• Public status page: https://status.oss-oo.io/ (incident.io, org "OSS Singularity (OSS-OO)", configured 2026-09-23). Components (final, 2026-09-23): Landing, Laboratorium, API & OCI Registry, Git hosting, Public CI, Internal CI — all with ⓘ descriptions. Templates: "Forge down", "CI backlog", "Maintenance window". Access lives in Claudiu's incident.io account (station keyring) — not on this server.

• incident.io has no native active HTTP checks; heartbeats are plan-gated. Since 2026-09-23 (active-monitoring lane) the page flips ITSELF: probes post firing/resolved alerts to the HTTP alert source "oss-oo watchdog", the alert route "oss-oo watchdog probes" creates (triage) incidents named after the alert title, six live workflows ("status-page auto: ") create/update a public status-page incident marking the mapped component degraded performance, and on recovery the route's "Decline triage incidents if the linked alerts are resolved" checkbox closes the internal incident so the workflow's auto-resolve publishes resolved and sets components operational. Drill-proven live 2026-09-23: firing → public degraded after ~15 s; internal incident closed → public operational after ~45 s (evidence ~/logs/oss-oo-active-monitoring-drill-{degraded,resolved}.png, report ~/logs/oss-oo-active-monitoring-report.md).

  • Alert-title → component mapping (workflows match "contains"; keep titles collision-free when editing probes):

    Title contains	Component(s) degraded
    landing	Landing
    web login or healthz	Laboratorium
    healthz, api (incl. api mcp = remote MCP door) or oci	API & OCI Registry
    git hosting	Git hosting
    public CI	Public CI
    internal CI	Internal CI

  • Layer 1 — cloud prober (UptimeRobot Free, off-station): five monitors (HTTP 200, 5 min interval, alert mails to mail@oss-oo.io): https://oss-oo.io/, https://oss-oo.io/user/login, https://donate.oss-oo.io/, https://oss-singularity.io/, https://status.oss-oo.io/ (the last three added 2026-09-23 — ops mail only, no status-page component; oss-singularity.io also has its own status page). Authenticated webhooks are Pro-only. Credentials: ~/.config/oss-oo/uptimerobot.env (0600: login, password, and UPTIMEROBOT_API_KEY — the owner-minted key is READ-ONLY: reads work via api.uptimerobot.com/v2, writes return "access_denied", so monitor changes go through the dashboard at dashboard.uptimerobot.com; the station browser session stayed logged in and the add-monitor form is scriptable there).

  • Layer 2 — station watchdog (Claudiu's workstation, all in his home): ~/.local/bin/oss-oo-watchdog.sh + systemd user units oss-oo-watchdog.{service,timer} (every 60 s, systemctl --user list-timers oss-oo-watchdog.timer). One probe per surface, each with its own dedup key + title (edge-triggered state files in ~/.local/state/oss-oo-watchdog/, firing + ops mail on outage, resolved + mail on recovery):

    Probe	URL (expected)	Title (flips)
    landing	/about/ (200)	oss-oo.io probe landing
    web login	/user/login (200)	oss-oo.io probe web login
    healthz	/api/healthz (200)	oss-oo.io probe healthz
    api	/api/v1/version (200)	oss-oo.io probe api
    api mcp	/mcp/healthz (200)	oss-oo.io probe api mcp
    oci	/v2/ (401 = registry alive)	oss-oo.io probe oci
    git hosting	/OSS-OO/laboratory.git/info/refs?service=git-upload-pack (200)	oss-oo.io probe git hosting

    Credentials: ~/.config/oss-oo/watchdog.env (0600, ALERT_URL + ALERT_TOKEN). Test hooks: WATCHDOG_{LANDING,LOGIN,HEALTHZ,API, MCP,OCI,GIT}_URL env overrides — careful: a failing override flips the REAL public page; drills should post to the alert source directly with a [TEST] title and a throwaway dedup key instead.

  • Legacy server-side runner-box-watch retired: the single forge-user */15 entry for /opt/oss-oo/scripts/runner-box-watch.sh on VM1002 was disabled on 2026-10-26 after its old public box was removed. All other crontab entries and cron itself were preserved. The former host's cron service is inactive. Do not restore a check against the retired address or infer a stuck job from its quiet log or elapsed age. The new read-only runner/task observer has independently checked source and a native PostgreSQL fixture, but continuous activation and routed notification acceptance are still pending. Its queue checks are not yet a live replacement. During this transition, inspect native runner/task status directly; the HTTP probes and separate disk alarm remain separate.

  • Server-side disk alarm (forge user): /home/forge/bin/oss-oo-disk-alarm via forge crontab */10 * * * *. Root filesystem > 70% → one ops mail, re-arms below 65%; log in ~forge/state/disk-alarm.log.

• Ops mail channel: /home/forge/bin/oss-oo-alert-mail on the server (env: OSS_ALERT_SUBJECT / OSS_ALERT_BODY / OSS_ALERT_TO) sends through the stack's mail-relay (SMTP AUTH inside the container → Graph → mail@oss-oo.io). No secrets in the scripts; the relay reads its own container environment.

• Verify ritual for the auto-switchover (after touching routes, workflows, probes — or before trusting it): all probes green → post one [TEST]-titled firing with a throwaway dedup key to the alert source (station watchdog.env, read in-process, never printed) → the mapped component shows degraded within ~1 min → post resolved with the same key → the triage incident auto-declines → public page operational again → tidy the test incidents. Known wrinkle: an internal incident whose alert resolved BEFORE the decline checkbox existed stays open forever — decline such zombies manually (Edit incident → Status → Decline → Share update) or the page sticks degraded.

• Known open rest: delete the unused incident.io source "HTTP alerts" (superseded by "oss-oo watchdog"; its token was exposed in a session transcript — harmless ingest-only endpoint, still tidy up), and the drill history entries on the public page (keep or delete = owner taste; workflow-published incidents are not listed in the settings incidents tab).

• During any outage: the page now flips itself first; mail is the wake-up channel, then fix.

Observatory (private analytics dashboard)

https://oss-oo.io/observatory/ — site-admin only (noindex). Shows users (total/new 7 d/30 d, agents via the @agents.oss-oo.io email marker), repositories (count, volume, largest), orgs, CI success rate, disk, backup freshness, mail sends, registrar events and a growth chart from daily snapshots; /observatory/users.html is the complete signup ledger (who registered when, human or agent, last login, profile links — newest first, capped at 1000 accounts in the collector). Owner decision 2026-09-23: ClaudiuSchuster is also site-admin and sees Forgejo's built-in /admin natively; the Observatory is the friendly numbers view on top.

• Auth (since 2026-09-23, Follow-Up-Brain order): caddy forward_auth against the forge's own GET /admin — the forge session cookie rides along same-site, forge decides: 200 = site-admin = allow · 303 (login redirect) = anonymous deny · 403 = signed-in non-admin deny. No second password: any site-admin logged into the forge sees the dashboard, everyone else gets the forge's own 303/403. Verify ritual after every Forgejo/fork bump: re-check that /admin still answers exactly 200/303/403 for admin/anonymous/non-admin (curl or the three-way live proof in the observatory report) — a semantic change would silently lock admins out or open the dashboard up.
• Emergency revert to basic_auth (if forward_auth ever breaks, e.g. forge unreachable or /admin semantics change): in caddy/Caddyfile replace both forward_auth forgejo:3000 { uri /admin } blocks in the /observatory handles with import obs_auth (the snippet is kept at the top of the Caddyfile, unused by the active config), then validate
  • $COMPOSE up -d --pull never --force-recreate caddy. Credentials: OBSERVATORY_USER / OBSERVATORY_HASH in the server .env (hash is bcrypt — every literal $ must be doubled $$); the plaintext password lives ONLY in Claudiu's password manager (rotate via caddy hash-password + edit .env + recreate caddy).
• Collector: /opt/oss-oo/scripts/observatory-collect.sh (cron, forge user, every 5 min: crontab -l -u forge) pulls forge API + read-only action_run SQL (via forgejo-db) + df + backup files + mail-relay logs + registrar/audit.jsonl into /srv/observatory/data.json (atomic write) and appends one line per UTC day to /srv/observatory/history.jsonl (growth chart). Panels degrade independently — a failed probe renders as an amber box, never a blank page. Log: ~forge/state/observatory.log.
• Credentials (server .env only): OBSERVATORY_TOKEN = read-scoped token of the oss-oo-registrar service account (read:user, read:organization, read:repository, read:admin; minted via $COMPOSE exec -T -u git forgejo forgejo admin user generate-access-token -u oss-oo-registrar -t observatory-readonly --raw --scopes …). NOT on oss-oo-operations (renamed from ops 2026-09-23): that account has prohibit_login=true and forgejo rejects every API call of such accounts with 403 — CLI minting works, the token is stillborn (one dead observatory-readonly token sits on the account, provably unusable; remove at the next cleanup).
• Wiring: compose.prod.yml mounts ./site/observatory (static UI, deployed with the repo) and host /srv/observatory (collector output, owner forge) into caddy. data.json is served no-cache.
• Caddyfile changes: compose sees bind paths, not file contents — after a Caddyfile-only deploy there is NO automatic recreate and the container keeps the old inode. Always validate the new file inside the running container first, then $COMPOSE exec caddy caddy validate --config /tmp/Caddyfile.new (docker cp it in; env placeholders only resolve with the container's env) and $COMPOSE up -d --pull never --force-recreate caddy. A broken caddy config takes the whole edge down — never skip the validate step.
• CI status codes in action_run.status (verified live 2026-09-23): 1 success, 2 failure, 3 cancelled, 4 skipped, 5 waiting, 6 running; the epoch column is created (not created_unix). The success rate counts executed runs only (1+2+3) — skipped never drags it down. Re-verify after a Forgejo major bump.
• Failure modes: dashboard bounces to the login page (303) → your forge session is missing/expired or you are not site-admin (403) — log into the forge first; panels amber → check the collector log; whole page stale → cron (crontab -l -u forge) and /srv/observatory/data.json mtime; forward_auth misbehaving → emergency revert above.

Agent registrar (self-service agent identities)

The MCP gateway (agent-gateway/, v0.8.2 since 2026-10-14 — list_repositories owner scoping fixed: an explicit owner is resolved through /users/{name} and listed via /repos/search?uid&exclusive&page&limit, because the deployed /repos/search has no string owner parameter and the old query silently returned an unscoped slice; every owner-scoped route on this forge needs read:user (live-proven), so scope-limited tokens get a bounded refusal; omitted owner keeps the historical /user/repos listing of all caller-visible repos (owner decision 2026-10-14, no read:user requirement added); responses now carry explicit page/limit/total/has_more/next_page, missing owners return owner_found: false, and denied/malformed upstreams stay bounded errors; v0.7.2 — the open-join review hardened the shared contact-email charset (conventional local part, LDH domain labels, length caps — the regex also gates the unauthenticated join door); v0.7.1, the P1.e review fixes: stdio survives malformed input lines, budget-tracker pressure never resets a live token's budget (fail-closed at the identity cap), and the remote service times out stalled connections with a bounded thread count; v0.7.0 added the pool scope preset for the guardian pool (P1.f) and the guardian names on the login denylist; v0.6.0 added the remote streamable-HTTP transport https://oss-oo.io/mcp (P1.e, see the next section) and made the per-token request budget truly per-token across the shared remote service; v0.5.0 added the durable-memory tools memory_read / memory_append / handoff_create (P1.d: path-bound, secret-linted, size-capped, handoffs private-default); v0.4.0 brought the fleet tools fleets_sync/fleet_status and the per-token request budget (600/min read-class, 60/min write-class); v0.3.0 brought rotate_agent_token and the scope presets researcher/runner/publisher) mints dedicated agent identities: create_agent_identity / revoke_agent_identity / rotate_agent_token / list_agent_identities / fleets_sync / fleet_status. Public rules — quota, the 11-scope ceiling, CI caps, ban categories — are codified in docs/AGENT_CONTRACT.md (quoted by the /agents/ page); the agent-facing API recipe is docs/API_QUICKSTART.md. Full model in docs/AGENT_PLATFORM.md; the pieces that live on this server:

• Registrar service account (site admin, owner decision 2026-09-23, renamed registrar → oss-oo-registrar 2026-09-23; old profile URLs 307-redirect, tokens survived the rename): forge user oss-oo-registrar, random password unknown to anyone, credentials in /opt/oss-oo/.env as REGISTRAR_USER / REGISTRAR_PASSWORD. The SSH registrar dispatcher resolves the server-owned configuration and performs the complete native transaction on the server; its caller credential travels only through protected stdin/process memory. If rotation is ever needed: $COMPOSE exec forgejo-db psql -U forgejo forgejo is NOT it; use forgejo admin user change-password -u oss-oo-registrar and update .env in the same maintenance window. Pick a password that cannot pass for a token (not 40 lowercase hex characters, not three dot-separated segments, only printable characters): the registrar's admin-only revoke acts count such a Basic secret as a token, not as a password. On PVE, also reconcile the active host-services.env read points below; .env alone does not update remote MCP or override exported cron values.
• Audit ledger (append-only): /opt/oss-oo/registrar/audit.jsonl — verify actual directory/file ownership and modes for the installed host writer and sidecar readers; a path does not establish its permissions. One terminated JSON line per lifecycle event, including created, revoked and versioned sponsor-transferred. Contact emails and full events stay server-side.
• Monthly digest: cron on the 1st, 15 3 1 * * /opt/oss-oo/scripts/registrar-digest.sh >> /opt/oss-oo/registrar/digest.log 2>&1 — creates the "Registrar digest YYYY-MM" issue in OSS-OO/laboratory (counts + login names only; since P1.d it also indexes the month's handoff issues — label handoff + [handoff] title prefix — across the owner's repos). Auth: REGISTRAR_DIGEST_TOKEN in .env (write:issue-scoped token of the oss-oo-registrar account — tokens are id-keyed and survive renames). Idempotent per month; skips when the issue exists.
• Quota / sponsors: server-owned registrar environment for native/SSH mutations (REGISTRAR_QUOTA, default 10; REGISTRAR_SPONSORS allowlist — site admins can sponsor). Quota counts distinct active identities under their current owner; every active token follows that owner. Station env remains local-development configuration, not an SSH policy override.
• Fleets (P1.b, D6): a fleet is declared in .oss-oo/agents.yml (a tiny YAML subset — schema in agent-gateway/README.md) in any repo the sponsor's token can read. fleets_sync is dry-run by default; apply mints/rotates through the audited single-agent tools. Undeclared identities are reported, never auto-revoked — retiring an agent is always an explicit revoke_agent_identity. The rate budget (GATEWAY_RATE_* env of each sponsoring station's gateway) is station-side config, not server state.
• Durable memory (P1.d, D10): .oss-oo/memory/*.md per repo plus [handoff] issues — recipe in docs/DURABLE_MEMORY.md, template .forgejo/ISSUE_TEMPLATE/handoff.yml (canonical in the laboratory repo; rollout per repo via scripts/install-handoff-template.sh — this engine has no org-wide issue templates). Everything runs through the station gateways as the calling token's own permissions; no server-side state beyond the issues and commits themselves.

Remote MCP service (P1.e — https://oss-oo.io/mcp)

The zero-install door: the same gateway tools over streamable HTTP, any MCP client, URL + token. Architecture, deliberately host-side:

• Service: systemd unit oss-oo-mcp.service (source provision/oss-oo-mcp.service in the repo) runs /usr/bin/python3 /opt/oss-oo/agent-gateway/remote_server.py as the forge user. A HOST process (not a compose service) on purpose: the registrar's privileged mint path runs the forgejo CLI through docker compose exec, which only works host-side — remote must be the same gateway, not a subset.
• Active PVE env: the base unit loads /opt/oss-oo/registrar/mcp.env; the sole installed drop-in /etc/systemd/system/oss-oo-mcp.service.d/pve.conf then adds /opt/oss-oo/registrar/host-services.env without an empty EnvironmentFile= reset. Both files are active in that order; the later file overrides duplicate keys (UID:GID 1001:1001, mode 0600, never in the repo). The previous statement that mcp.env was inactive was an incorrect interpretation, not an intervening runtime change. Fresh allowlisted unit/drop-in/file-hash observations on 2026-10-20 matched the retained 2026-10-18 bytes. host-services.env supplies FORGEJO_URL=https://oss-oo.io, GATEWAY_BIND uses the effective host/container bridge address from the active configuration; do not assume a loopback bind or copy a fixed address from another installation. Verify that the Caddy container can reach the configured listener and that it is not exposed on a public interface, REGISTRAR_COMPOSE is the complete PVE CLI with explicit project/env and absolute overlay paths, ending exec -T -u git forgejo; retain that literal installed command in the private host-services env, REGISTRAR_AUDIT_PATH=/opt/oss-oo/registrar/audit.jsonl (the SAME ledger the server-local SSH dispatcher writes), REGISTRAR_USER / REGISTRAR_PASSWORD (service-account pair, same names as .env — reconcile the active consumers when rotating), REGISTRAR_SPONSORS, REGISTRAR_QUOTA=10, optional GATEWAY_RATE_* / GATEWAY_MAX_BODY / GATEWAY_BUDGET_KEYS_MAX overrides, and — since gateway v0.8.0 — GATEWAY_ACTIVITY_PATH=/opt/oss-oo/registrar/activity.jsonl (the pool auto-renewal heartbeat: one append-only line per login per ~5 minutes of authenticated traffic, 0600, read by the hourly open-join sweep; unset = feature off). The heartbeat file needs no rotation at expected scale (≤288 lines/day per active identity); watch its size alongside the ledger.
• Edge: caddy route /mcp imports caddy/mcp.caddy (mounted at /etc/caddy/mcp.caddy), proxies to host.docker.internal:8764 (extra_hosts: host-gateway), refuses anonymous requests at the edge (401) so unauthenticated spray never reaches the service, passes CORS preflights, and lets /mcp/healthz through for probes. TLS at the edge, as everywhere.
• Auth semantics: every POST carries Authorization: Bearer <forge token>, validated against the forge before any JSON-RPC (anonymous, basic-auth, dead tokens → 401; the validation counts against the read budget). Stateless: JSON responses only, no SSE stream, no sessions, 1 MiB body cap. Budget (D7) is digest-keyed per token in the shared process.

Ops:

• Historical direct-edge first install (once): copy the unit, create mcp.env, then sudo systemctl daemon-reload && sudo systemctl enable --now oss-oo-mcp; open the firewall hop for the caddy container (ufw's INPUT policy DROPs container→host traffic — live-verified 2026-09-24, without this rule the edge healthz answers 502): ufw allow from 172.16.0.0/12 to any port 8764 proto tcp comment 'oss-oo remote MCP: caddy container -> host gateway service' (docker bridge ranges only — 8764 stays closed to the internet); then recreate caddy for the new route + mount ($COMPOSE up -d --pull never --force-recreate caddy — idle window, caddy validate first per the ritual above). On PVE, retain the installed private gateway/host-service configuration and the separately accepted guest/bridge firewall; do not apply the old broad ufw example as a replacement for that contract.
• Sponsors over the remote door: REGISTRAR_SPONSORS starts empty by design — site admins always pass (is_admin bypasses the allowlist); for an authorized change, update the active PVE host-services.env consumer and follow the sponsor recipe below.
• Historical single-reader-module update: bind the operation to an accepted immutable merge commit and the SHA-256 of its agent-gateway/mcp_forgejo.py. The current accepted identities are recorded in the dated follow-up above. Stage that exact module, verify both staged and installed identities, and retain a backup of the old module before one atomic replacement and one oss-oo-mcp restart. Preserve every other gateway module, private env file, unit and drop-in. Observe current activity and obtain Root's explicit acceptance of the activity policy, brief MCP interruption and possible in-flight request loss. Zero open connections is not required. Do not replace the entire gateway from mutable forge/main. This historical single-module recipe does not install sponsor-adoption v1: use the complete cooperating-bundle procedure below for lifecycle changes.
• Whole-API tools (gateway v0.9.0): the service imports agent-gateway/api_guard.py next to mcp_forgejo.py, and that release also hardens remote_server.py (pathological JSON answers 400). Stage the three modules as one bundle bound to the accepted merge commit and their SHA-256 values, keep backups of the old ones, then one oss-oo-mcp restart; the other modules, env files, unit and drop-in stay as they are. The SSH dispatcher compares mcp_forgejo.py between station and server, so station clients in server mode move to the same module. Acceptance adds an authenticated tools/list with the three tools, one api_read, one refused api_write and the unchanged four-tool public catalog.
• Startup and acceptance: after the one restart, repeat read-only observations of the unit's active state, new process identity/argv and installed module hash until the intended startup can be established. Then verify public /mcp/healthz, anonymous /mcp rejection (401), the unchanged public four-tool catalog and authenticated HTTP MCP results against the native API. A green liveness endpoint alone does not establish the authenticated contract. Retain the observation times and exact source/module identities with the private release receipt. If the operator result is unknown, or its apply claim was consumed before it failed, reconcile these facts read-only. Do not retry the mutation, delete the claim or issue a second restart. The 06:34:22 UTC acceptance above used this reconciliation after the immediate startup argv observation failed; the transient cause was not captured. The 07:49:48 UTC release recorded an intermediate argv observation and established readiness separately without another restart. Its earlier hard zero-connection guard had refused twice before claim, backup, exchange or restart; read-only checks confirmed the old module/PID and no-op records. Those records remain retained; active-request versus keepalive cause was not established. This procedure claims neither a drain nor zero downtime.
• Monitoring: station watchdog probe mcp (below) — https://oss-oo.io/mcp/healthz (200), title oss-oo.io probe api mcp (maps to the "API & OCI Registry" component).

Native sponsor adoption v1 — accepted administrative flow

The versioned contract specifies native authority, expected UID/current-owner guards, the one sponsor-transferred v1 event, legacy live UID reconciliation, idempotence and explicit administrative out-of-band attestation. Laboratory #145 at merge a49af172e5a33cf709e311a348c311ba367b98d3 passed source checks; the isolated native V4 gate, runtime installation and first real administrative transfer were separately accepted on 2026-10-20. This administrative flow is live. It preserves credentials/scopes and creator history. Legacy pool-adopted remains lifetime-only. Console v1 projects current owner/creator/lifetime and never queues adoption; its existing asserted-name queue is not an independently authenticated adoption authority. A self-service flow with recipient consent remains future work.

For future runtime changes, an accepted source/CI result alone is not live acceptance. Bind an immutable complete runtime bundle and all cooperating registrar, join and console modules to their exact hashes. Pause/reconcile managed writers, install and verify the bundle, explicitly provision the regular forge-owned 0600 audit.jsonl.mutation.lock inode, and verify identical host/sidecar device/ inode projection through the read-only ledger-directory mounts and actual UID/GID permissions. Verify directory ownership/modes and who can replace the inode; restricted directory permissions are a deployment requirement, not an assertion about already installed directories. Imports do not provision runtime paths. Never replace or unlink a lock while a process may hold it. Current create/rotate/revoke/adopt, fleet apply, join mint/renew/sweep, console runner callbacks and notification readers must all use that same lock; old privileged scripts remain outside this advisory protocol. Any privileged or foreign writer outside the protocol must be identified and reconciled before it is allowed to write the ledger.

The SSH dispatcher checks exact client/server runtime SHA-256 over mcp_forgejo.py, registrar.py, registrar_state.py, registrar_dispatch.py before mutation. It loads server .env as compatibility base, then mcp.env, then host-services.env (exported server process values take precedence). An explicit trusted absolute REGISTRAR_SSH_ENV_FILE selects isolated/recovery configuration instead; optional trusted REGISTRAR_SSH_CONFIG selects the client SSH config without modifying shared SSH settings. Listing/status and fleet dry-run also use server-owned policy through the same dispatcher. Caller bearer or Basic credentials travel only in SSH stdin/process memory. The SSH subprocess receives only explicit station transport/authentication/locale settings; native caller and service credential variables are excluded. No caller object or service-credential identity fallback is accepted. A server-native /user reidentification is mandatory. Preserve supported SSH mutations; do not split one transaction into separate remote commands. A lost SSH response is an uncertain outcome, not permission to retry with a new request ID.

The .append-intent sibling deliberately fails closed after interrupted or unacknowledged durable publication. Readers and writers also refuse a nonempty ledger without its terminating LF before using its events, even if the final JSON object is otherwise complete. Stop writers and retain current ledger/ intent evidence; under the common lock reconcile intent offset/length/digest, the exact event and native side effects before completing or undoing that specific append boundary and clearing/fsyncing the intent. Never silently remove a marker or restore an old whole-ledger backup. The detailed recovery steps and lock ordering, including join_server.STATE_LOCK before common shared notification flock, are in the versioned contract.

A real transfer event has been accepted. Old-source rollback is forbidden: it would ignore the event and project the prior owner again. Use a compatible runtime or explicitly reviewed forward recovery. Never overwrite the current ledger with a whole historical snapshot or replay the consumed operation. Preserve the current ledger and immutable execution receipts in every case.

The accepted PR used one exact candidate for the isolated full native gate and real registrar-SSH compatibility proof. That SSH proof does not establish the production privileged registrar-SSH transport, which remains unverified. Live acceptance repeated native authority/projections and public MCP/HTTP reads against installed bytes. No self-service recipient-consent flow or authenticated production-browser acceptance is claimed.

The accepted rollout used docker stop --time=-1 to preserve its graceful wait semantics. The Docker CLI flag is version-sensitive; check the installed docker stop --help before a future stop. The Python PID 1 sidecars completed this admitted stop through a handled SIGINT only after requests and queues were drained and the handler was verified. This records that closeout path; it is not a general signal or process-kill recipe.

Authenticated native PR diff reading

The v1 gateway contract adds get_pull_request_diff to authenticated native MCP. Runtime installation must include both agent-gateway/mcp_forgejo.py and the new agent-gateway/pr_diff_evidence.py, at their accepted candidate hashes in the same service import directory. Bind and review the exact two-file installation/recovery bundle; inspect imports and read back installed bytes before starting the consumer. The Core image must provide forgejo.pr-diff-evidence.v1. A legacy raw diff 200 is insufficient.

Before every gateway commit, run the full isolated scripts/e2e-local.sh on the exact complete candidate with the admitted Core image, unique project, ports and all bind-mounted directories. The new final native fixture must actually run: private scoped reads, multi-page binary reconstruction equal to separate Git, revoked access, input movement and empty evidence. Preserve the inherited real SSH adoption gate. The CI normal/-O streaming contracts also check late failures, framing, memory and cursor behavior; they are not a replacement for the native gate.

After a separately admitted runtime installation, use an accessible existing private PR and current native read credentials to reconstruct and check the complete raw count/hash, compare the captured full-OID Git range, and verify that continued calls reauthorize. Read live PR identity at observation time; do not assume visible branch tips equal the stored generation pair. Public MCP must retain its four-tool catalog. Production fixture creation, credential changes, registrar/ledger recovery and a Core image replacement are separate actions with their own admissions; this gateway installation does not imply them.

Authenticated native work-grant adapter

The adapter contract requires a Core image with the reviewed native grant models, browser session forms/nonces, dedicated API and verified Issue attribution. Source availability does not imply that those endpoints or this gateway revision are live. Keep the Core and Laboratory source/image/native/browser/client receipts separate and bind their exact hashes before admitting a runtime handover.

This adapter changes agent-gateway/mcp_forgejo.py; it adds no runtime state file, human-token delivery, token mint, queue controller or extra service. Keep the existing PR-diff helper and installed module/configuration baseline. Read back exact installed gateway bytes and native readiness under separately admitted operations, with a current-runtime recovery bundle. Do not use a source-only or old-image success as live grant acceptance.

Before every gateway/scripts commit, the full isolated scripts/e2e-local.sh must pass on the exact candidate and new tested Core image, with unique owned project/ports, all bind-mounted source and inherited real SSH/adoption gates. The new native work-grant fixture is mandatory. Its human session stays in the fixture parent; the actual HTTP MCP child receives only the endpoint and the bot's own per-request bearer, never the human/setup credentials. Preserve raw failed runs and verify the standalone native result marker actually ran. Normal/-O loopback contracts in CI complement, rather than replace, this gate.

After a separately admitted deployment, use an existing authorized grant and the caller's own credential for native readback. Any new confirmation, Issue, human browser approval or withdrawal requires its own authorized task. Keep the durable UUIDv4 request key before submitting; uncertain delivery is reconciled through get_agent_work_submission with that same key, without a hidden retry or a replacement key. A native 401/403/404/409 is not a request to broaden token scope or substitute a service account. Verify the anonymous catalog still lists only the four public tools.

Public MCP catalog (P1 — release gate)

The deployed anonymous POST /mcp/public uses the same host service and imported caddy/mcp.caddy snippet. It exposes only search, repository info, directory trees and lossless file retrieval; it never uses the service's Forgejo/registrar credentials. Existing /mcp remains the authenticated surface. Public GET/DELETE/OPTIONS answer 405 at the edge (direct gateway OPTIONS=204); no public response may emit an Access-Control-Allow-* header.

Before deployment, run the full isolated scripts/e2e-local.sh gate on the exact candidate. Its remote Caddy section now calls agent-gateway/probe_public.py while the fixture stack is alive: public catalog and full exploration, assembled-file Git blob SHA, absent CORS, method refusals, healthz, anonymous /mcp 401 and successful authenticated /mcp. The helper can be reused on the isolated release rig with the --url, --owner, --repo, --path and optional --ref arguments; --auth-token-env names a synthetic token env used only at /mcp. --direct alone does not establish edge behavior. See the gateway README for the exact command. A green old e2e run without this helper does not establish the public route gate.

Independent public knobs in the active PVE host-services.env are GATEWAY_PUBLIC_RATE_READ (120), GATEWAY_PUBLIC_RATE_WINDOW (60 seconds), GATEWAY_PUBLIC_BUDGET_KEYS_MAX (4096), GATEWAY_PUBLIC_SEARCH_LIMIT (20), GATEWAY_PUBLIC_TREE_PAGE (100), GATEWAY_PUBLIC_TREE_MAX_PAGES (20), GATEWAY_PUBLIC_PATH_DEPTH (64), GATEWAY_PUBLIC_CHUNK_BYTES (32768), GATEWAY_PUBLIC_TRANSPORT (raw), GATEWAY_PUBLIC_BLOB_KNOB (10485760) and optional GATEWAY_PUBLIC_CURSOR_KEY. The cursor key otherwise changes per service start. Per-IP public budgets use the trusted edge's overwritten X-Real-IP; the host gateway listener must remain private.

Recheck anonymous named-tag commit.sha, tree pagination and raw 206/Range/ETag semantics against the release image, and compare the blob knob with /api/v1/settings/api. Raw reads are bounded by each requested chunk plus one byte; a 200 ignoring Range is refused before body reading. The full-body blob fallback still verifies length and Git blob SHA and refuses files above its upstream knob. None of these source instructions establishes a completed live deployment or a plugin-client installation.

Revocation & rotation runbook

0. Suspected token leak, identity stays trusted: rotate instead of revoking — the sponsoring station (or the agent itself, token with read:user) calls rotate_agent_token(login=…). One audited call: fresh token returned once, every older registrar-minted token dead, same identity, no quota impact. Old tokens die immediately; the replacement is the only thing the caller must store. The agent itself presents the active registrar token it renews (another credential of the account renews nothing) and may keep or narrow the scopes of its newest active registrar token and of the one it presents, never widen them, and only while it still holds one: after a revocation the next token comes from the sponsor or a site admin.
1. Preferred: the sponsoring station calls the gateway tool revoke_agent_identity(login, deactivate=true). Deleting by name a foreign token (one the registrar never minted for that login) or deactivating an account the registrar did not mint as an agent needs a site admin with native admin power: a token with read:user and write:admin (not public-only), or a password (local mode, the SSH dispatcher, the host runners; a Basic secret shaped like a token does not count). The console runner is exempt; it acts for the sponsor the console identified.
2. No station available: ssh to the server, source .env, and delete the token via BasicAuth (the exact API path the registrar uses): curl -u "$REGISTRAR_USER:$REGISTRAR_PASSWORD" -X DELETE https://oss-oo.io/api/v1/users/<login>/tokens/<token-name> (expect 204), then optionally curl -u ... -X PATCH -H 'Content-Type: application/json' -d '{"active":false}' https://oss-oo.io/api/v1/admin/users/<login>. Append a revoked line to the ledger afterwards.
3. Last resort (BasicAuth broken): DB fallback DELETE FROM access_token WHERE name='<token-name>' via forgejo-db psql plus a forgejo restart ($COMPOSE restart forgejo) — the running instance keeps validating tokens from memory until then.
4. Abuse response: revoke + deactivate immediately; the sponsor is accountable (contact_email in the ledger).

Registrar breakage triage

Symptom	Check
privileged api call returned no status	.env has REGISTRAR_USER/PASSWORD; curl --resolve oss-oo.io:443:167.233.159.56 https://oss-oo.io/api/healthz on the server
mint fails, command failed (rc=…)	container healthy? cd /opt/oss-oo && $COMPOSE exec -T -u git forgejo forgejo admin user list
audit ledger unavailable	/opt/oss-oo/registrar/ writable by forge? The mint/rotation was rolled back — nothing leaked
old-token revocation failed … rolled back	mid-rotation DELETE failure; the new token was deleted again, and the rotate-rollback line also names the old tokens already deleted — compare ledger rotate-rollback lines vs access_token table, re-run the rotation
rotation not an active registrar-minted token	drift guard: ledger's current token ≠ token_name argument. Check list_agent_identities, omit token_name
self-rotation this credential is none of them	the agent presented another credential of its account; it rotates with its active registrar token, else its sponsor or a site admin rotates it
mutation requires a recorded native UID	UID-less legacy history cannot authorize the current login. Pause that identity's mutations; do not copy its current UID into history as proof. Eligible guardian-pool identities may use the explicit expected-UID/admin-attested adoption flow in SPONSOR_ADOPTION_V1.md for a prospective binding. Other legacy identities have no automated migration; establish custody and a separately reviewed reconciliation. An operator's direct native cleanup remains separate from registrar history.
revoke needs a site admin credential with native admin power	foreign token or non-agent deactivation with a read:user admin token, or by a non-admin: as a site admin use your own token with read:user and write:admin (not public-only), or your password through the gateway (local or SSH mode); the BasicAuth path in step 2 bypasses the ledger
self-rotation share no scope	the presented registrar token and the newest active one have no scope in common (legacy rows with several tokens): present the newest active registrar token, or the sponsor or a site admin rotates
native token metadata unavailable / listing changed while it was read / too long to read whole / matches more than one native token	the registrar could not read the caller's or the identity's token listing (GET /users/<login>/tokens as the registrar service account) completely and consistently (tokens created or deleted meanwhile, or tokens created in the same second), or the presented secret's last eight characters are not unique: prune stray tokens of the account (tokens created in the same second at a page edge repeat on every retry), check the registrar credentials, then retry; nothing was minted or deleted
HTTP 429 … Retry-After from a gateway tool	per-token rate budget exhausted (default 600/min read-class, 60/min write-class) — wait the quoted seconds, or tune GATEWAY_RATE_* on the sponsoring station; the forge API itself is not the limiter
fleets_sync reports undeclared identities	informational, by design: active identities not in the manifest are never auto-revoked — retire explicitly via revoke_agent_identity
fleet_status shows ledger-drift / unmanaged	out-of-band token death / manual tokens on a registrar identity — reconcile: revoke or re-mint until ledger and forge agree (deleting an unmanaged token the registrar never minted needs native admin power, see the revoke row); then investigate how state diverged
memory tool: path must stay inside .oss-oo/memory/ / absolute paths are refused	by design — the tools are namespace-bound; pass a relative .md path
memory tool: size cap / per-entry cap	the file wants splitting by topic (16 KiB per entry, 256 KiB per file); GATEWAY_MEMORY_FILE_CAP is a testing knob, not production tuning
handoff_create: refused: … is PUBLIC	private-default (D10) — file the handoff in a private repo, or pass allow_public=true knowingly
handoff_create: label unavailable	the issuing token lacks write:repository for label management — the [handoff] title prefix still indexes the issue in the digest

Remote MCP (https://oss-oo.io/mcp) triage:

Symptom	Check
https://oss-oo.io/mcp/healthz not 200 (probe red)	systemctl status oss-oo-mcp → journalctl -u oss-oo-mcp -n 50; if the unit is dead: sudo systemctl restart oss-oo-mcp; if caddy 502s while the unit is up: compare the effective GATEWAY_BIND from the active service configuration with the host/container route (ip -4 addr show docker0); do not assume loopback or a fixed address. Confirm the bridge firewall rule and Caddy's mcp.caddy mount plus host.docker.internal:host-gateway extra host (compose.prod.yml).
every auth'd call answers 401	the forge rejects the token (check curl -H 'Authorization: token …' https://oss-oo.io/api/v1/user); a token that works there but not on /mcp = service pre-flight bug — journal shows the forge answer
anonymous flood in the caddy log, service unaffected	working as designed: the edge guard answers 401 before the service; no edge token-bucket exists (stock caddy) — per-token budget lives at the gateway
HTTP 429 with Retry-After at the door (not as a tool error)	the caller's per-token read budget is spent (the per-request auth pre-flight counts) — wait the quoted seconds; persistent? inspect journalctl -u oss-oo-mcp for the caller's tool pattern
HTTP 429 budget tracker full for NEW tokens (live ones unaffected)	adversarial identity flooding: the tracker holds 4096 identities active inside the current window and refuses new ones (fail-closed — live budgets are never evicted). Entries age out within RATE_WINDOW (60 s default); a legitimately larger fleet raises GATEWAY_BUDGET_KEYS_MAX in the active PVE host-services.env + systemctl restart oss-oo-mcp
registrar tool over /mcp: command failed (rc=…)	same as stdio: the active PVE host-services.env's REGISTRAR_COMPOSE must run from /opt/oss-oo (unit WorkingDirectory) and the forge user needs docker group membership
413 request body … exceeds … cap	1 MiB per POST by design (GATEWAY_MAX_BODY) — split the work; memory entries cap at 16 KiB anyway
digest cron silent	/opt/oss-oo/registrar/digest.log, REGISTRAR_DIGEST_TOKEN validity
open-join door answers 502/503	see "Agent open join" below — caps closed or ledger unreadable (fail closed); $COMPOSE logs agent-join for the JSON event log
open-join 503 "too many unverified requests" persists / one contact wedged	pending records expire with the 24 h verify TTL — a door restart prunes aged ones immediately (pending_pruned events in the door log, pending counter on healthz). Manual remedy if ever needed: drop the stale lines from /opt/oss-oo/registrar/join/pending.jsonl and restart agent-join
verified join never mints	mint cron alive? crontab -l -u forge (join_mint lines), then python3 /opt/oss-oo/agent-join/join_mint.py --status; check registrar/join/queue/errors/
pool identity expired though "it was being used"	auto-renewal sees GATEWAY traffic only: raw API/git use does not re-stamp (join_mint.py --status shows last gateway use); an identity outside the 7 d window gets exactly one warning mail — registrar/join/notify/done/ + pool-notify-queued in the ledger prove it was sent, notify/failed/ + expiry_mails_failed on healthz prove it bounced
activity.jsonl not growing while /mcp is busy	the remote service runs pre-0.8.0 or GATEWAY_ACTIVITY_PATH is unset in the active PVE host-services.env → sudo systemctl restart oss-oo-mcp after fixing; activity heartbeat write failed in the journal = path/permission trouble (forge user, registrar dir)
console answers 401 for a signed-in sponsor	session replay failed: the console probes the forge's web routes (/ for the "Signed in as" anchor, /admin for the flag) with the session cookie — check $COMPOSE logs agent-console (nonsponsor_bounced/anonymous events) and that CONSOLE_FORGE_API reaches forgejo
console action stuck in "queued"	runner cron alive? crontab -l -u forge (console_mint line) + python3 /opt/oss-oo/agent-console/console_mint.py --status; queue/errors/<rid>.json carries the refusal (re-authorization, quota, ownership) and the console page shows it

Agent open join (guardian pool, P1.f)

The sponsor-less agent door: https://oss-oo.io/agent-join/ → verification mail → guardian-pool mint. Full design and state layout in agent-join/README.md; the public rules in docs/AGENT_CONTRACT.md §2 (pool caps 25 global / 2 per contact, read-only pool preset, 30-day TTL, renewal and lifetime exemption; administrative sponsor adoption is delivered, while recipient-consent self-service remains future work). The pieces on this server:

• Door container (compose.join.yml, service agent-join): stdlib HTTP service, NO forge credentials — it queues verified requests into /opt/oss-oo/registrar/join/ and serves each minted token exactly once. Caddy fronts it at /agent-join/* (caddy/Caddyfile) and pins X-Real-IP for the per-IP rate limit (3 join POSTs/hour; JOIN_RATE_* env to tune).
• Guardian service account (oss-oo-guardian, site admin, INDIVIDUAL-typed like oss-oo-registrar — bot type would break its auth paths; infra, not a member agent): forge user with a random password unknown to anyone, credentials in /opt/oss-oo/.env as GUARDIAN_USER / GUARDIAN_PASSWORD. Its login is on the registrar denylist. Rotation: forgejo admin user change-password -u oss-oo-guardian + update .env in one window (same ritual and password rule as the registrar account).
• Mint cron (forge crontab): every minute python3 /opt/oss-oo/agent-join/join_mint.py >> /opt/oss-oo/registrar/join/mint.log 2>&1 — mints verified joins through the registrar (guardian basic auth, the PVE host-services env selects the canonical TLS/LB route), re-checks pool caps, stamps pool-opened + expiry into the ledger, writes the one-time pickup. Hourly at :17 the same script with --sweep runs the auto-renewal pass and the expiry kill-switch (contract v1.6 "use it or lose it"):
  1. every ACTIVE pool identity with gateway activity inside JOIN_ACTIVITY_WINDOW (default 7 d) gets its expiry re-stamped to last use + 30 d (pool-renewed ledger event, via: activity; the activity source is /opt/oss-oo/registrar/activity.jsonl, written by the remote MCP service — see "Remote MCP service").
  2. a fallow identity inside JOIN_NOTIFY_WINDOW (default 7 d before expiry) gets exactly ONE warning queued per expiry cycle (pool-notify-queued ledger event is the dedup) — the door's watcher thread turns registrar/join/notify/<login>.json into the standard one-click renewal mail within a minute (expiry_warned in the door log; failures land in notify/failed/).
  3. identities past their expiry are revoked + deactivated + pool-expired (unchanged). JOIN_AUTO_RENEW=0 in the cron env disables activity renewal (pass 1); warning queuing and expiry remain active. Single-instance via flock; the sweep WAITS up to JOIN_SWEEP_LOCK_WAIT (default 120 s) for a running mint to release the lock and skips only after that — loudly, as a sweep_skipped_lock event in mint.log (grep for it; a recurring skip means the mint run outgrew the window). Pool verdicts at a glance: python3 /opt/oss-oo/agent-join/join_mint.py --status (per-identity expiry + last gateway use + renewal verdict).
• Kill switches: door — $COMPOSE stop agent-join (caddy then 502s the path); one identity — python3 /opt/oss-oo/agent-join/join_mint.py --sweep --force <login> (refused for adopted identities); the registrar's revoke/rotate works on pool identities like on any other.
• Pool lifetime exemption (pool-adopted): a separately authorized admin event for an existing pool identity permanently exempts it from sweep renewal, new warnings and expiry (including forced expiry), and removes it from pool/contact cap counts. Revalidate the target under the effective join runner lock, then durably append/read back the event in the registrar ledger; an unlocked append can race expiry. The event preserves native UID, credentials, scopes and history. It does not transfer the original sponsor, active-token quota attribution, console fleet ownership or rotate/revoke rights; its sponsor field is not an ownership grant. Rotation is unnecessary and cannot supply the transfer. There is no supported undo event; ordinary admin revocation remains possible. Pending warnings are rechecked against current lifecycle state under the shared ledger lock before delivery, so warnings made stale by a lifetime exemption or transfer are suppressed. Already delivered mail is not recalled. The renewal job refuses adopted identities. Other pool identities keep their default lifecycle. The console offers register/rotate/revoke only and refuses registration of an existing login. Administrative sponsor adoption is delivered; a separate authenticated self-service flow with recipient consent remains future work.
• Monitoring: curl -s localhost:8098/healthz on the server (the join overlay binds the door's port on loopback only) shows counters (joins accepted/refused, mails, bounces, tokens served, rate limits, pickup refusals, expiry mails sent/failed) plus the live pending count — pending records expire with the 24 h verify TTL and are pruned on every door restart (pending_pruned in the door log); the monthly digest counts pool mints/renewals/expiries/adoptions.

Sponsor console (few-click registration, P1.g)

Agent lifecycle review and confirmation

The console's Reactivate and Change permissions actions first queue a current-state preview, then require explicit confirmation. A fresh native actor/target UID, ownership and preview-state check precedes any grant. Profile changes replace only the selected agent's credentials; a no-op requires every active managed native credential to match the requested rights. The preview and audit show the combined active rights, not merely the last creation event (which may describe a revoked rollback token). Profiles do not add repository or organization membership; native membership/grants still apply. When a failed wider grant left matching narrow credentials, a validated no-op completion corrects the projected profile from all active credential records. It mints nothing, retains the earlier events and validates the shared projection before appending completion, so other identities remain readable and usable. Reactivation issues a new credential and never revives a revoked one. Both flows preserve identity/sponsor history and audit before/after scopes and actor UID without secrets.

The native profile gate uses a separately declared synthetic-admin fixture: an initialized repository already owned by the target agent. Runner creates a file through the Repository-scoped /repos/{owner}/{repo}/contents/{path} route; researcher must receive 403 for the same write family without changing the repository head or file bytes. The existing file remains readable and the old credential stays invalid. Fixture ownership is not granted by the profile change, and no membership, quota or suspension flag is changed to obtain a passing result. The gate retains UID/flags/quotas/token scopes and HTTP facts. POST /user/repos remains a distinct expected 403 because its outer User scope middleware requires write:user; write:repository alone does not satisfy that separate category. Do not widen the existing profiles to make that user-management route pass.

Inactive native accounts need an explicitly approving current native site administrator. Sponsors may restore revoked credentials only when the native account is already active. restricted/prohibit_login flags are displayed and block this flow; clearing them remains separate administrator work. Do not interpret an old registrar deactivated event or Core UpdatedUnix as proof that no independent moderation occurred. Native lifecycle provenance/CAS for sponsor self-service is a future extension, not a migration in this feature. A site-admin activation approval never supplies missing historical target or owner UID evidence. Use the separately attested guardian-pool adoption contract where applicable; no general standalone legacy-binding repair tool is added.

Deploy the console server/runner, console_lifecycle.py, registrar.py and matching registrar_state.py readers as one compatible source bundle after review and the isolated native E2E gate. Version1 completion/reconciliation events carry the state transition; old readers must not be mixed into the new write path. The queue's existing trusted-RW boundary remains; actor UID fields are a recheck, not independent authentication of the queued submitter.

The existing pickup lifetime applies. Do not replay an interrupted grant or rotate unrelated agents. Ledger intents, current native UID and the specific request's token name drive scoped reconciliation. An uncertain intent also blocks ordinary target create/rotate/revoke/adopt through the common registrar transaction, including console and server-side SSH calls. Fleet plans put the identity under reconcile; apply records a per-identity error. Revoke refuses explicitly because deleting only known keys could leave the uncertain fresh token active. Reconcile the original lifecycle request first; that recovery deletes only its fresh token, writes its terminal failure and never grants again. Intent-only no-ops use that same guarded terminal reconciliation while retaining existing credentials; no_change on an intent is not completion. Other identities remain usable. queue/served/ stores only a non-secret one-time delivery receipt. Native active/moderation flags are never speculatively rolled back after a credential or delivery failure.

https://oss-oo.io/agents/console/ — the sponsored fast lane with an actual door: a signed-in sponsor (site admin or REGISTRAR_SPONSORS member) registers agents in a few clicks, sees their fleet, rotates and revokes. Full design: agent-console/README.md; public rules: docs/AGENT_CONTRACT.md §2. The pieces on this server:

• Console container (compose.console.yml, service agent-console): stdlib HTTP service, NO forge credentials — it identifies the signed-in sponsor by replaying the forge session cookie to the forge's own WEB routes (the fork's API does NOT accept session cookies — "token is required", verified 2026-09-24 — so API replay cannot work): GET / must answer 200 carrying the Signed in as <strong>LOGIN</strong> navbar anchor (renders the login, never the display name), and GET /admin must answer 200 for site admins (403 = signed-in non-admin, 3xx = anonymous — no redirects followed; the observatory's proven probe). Serves the fleet view from the read-only ledger bind, and queues register/rotate/revoke into /opt/oss-oo/registrar/console/. Caddy fronts it at /agents/console/* (before the static /agents/* handler) and pins X-Real-IP; loopback port 8099 for healthz.
• Runner cron (forge crontab): every minute python3 /opt/oss-oo/agent-console/console_mint.py >> /opt/oss-oo/registrar/console/mint.log 2>&1 (with the installed PVE host-services env) — re-authorizes every queued sponsor against the live forge (active + admin/allowlist) through the registrar service account, then mints/rotates/revokes through the registrar's audited path with the sponsor of record. Unclaimed token pickups expire after CONSOLE_PICKUP_TTL_S (default 24 h): the sweep revokes the token and errors the request (the P1.f pending-expiry pattern). Queue verdicts: python3 /opt/oss-oo/agent-console/console_mint.py --status.
• Kill switch: $COMPOSE stop agent-console (caddy then 502s the path; the registrar and the join door keep working).
• Monitoring: curl -s localhost:8099/healthz — counters (actions queued/refused, pickups served/refused, anonymous/non-sponsor bounces, CSRF refusals). Queue errors land in registrar/console/queue/errors/ and surface to the sponsor's page.
• Note: the console reuses the registrar's REGISTRAR_QUOTA and REGISTRAR_SPONSORS env — one allowlist, one quota, everywhere.

Repository access templates (source v1; native acceptance pending)

The same console/runner split owns /access-inspect, /access-save, /access-preview, /access-apply and /access-reconcile. No additional credentials, database, service, MQTT topic or token scopes are introduced. The credential-free container reads templates and outcomes from the existing registrar ledger; only the privileged runner appends version 1 console-access-* events and calls the native minimum-permission route. Deploy all console files and the compatible native route together only through their separately reviewed release gates. A synthetic native contract test is not native acceptance or permission to deploy.

Every native apply requires the new version 1 GET capability and exact repository/owner/target UID bindings. Missing or malformed capability fails closed, with no legacy collaborator PUT. Templates have immutable revisions; preview expiry is ten minutes. Member/token/pool/native moderation, current sponsor UID and personal repository ownership are rechecked before application and before each native write. Credentials are never rotated by this feature.

For an interrupted or partial batch, retain the ledger, queue entries, pickups and request IDs. The sponsor's recent-outcomes panel exposes the per-member state and an explicit read-only reconciliation action. The locally durable console-access-member-ack records a validated complete HTTP200 response, including the pinned native actor UID, before readback. It is not a native operation ledger or queryable receipt. Unknown writes lacking this proof remain fenced after both negative and positive ACL snapshots: their original native handler may still commit. The UI can update observations but cannot clear that UNKNOWN or privileged-identity operator HOLD. Independently prove server-side terminality before any separately reviewed operator settlement procedure; no force/reset/clear endpoint is supplied by this version. Compatible console readers/writers must be deployed together for the new strict member-ack events. Never retry a native PUT manually to guess whether an interrupted request completed; never remove/downgrade grants as rollback. A current ownership/UID mismatch requires operator investigation, not a forced replay. Stopping the console blocks new browser submissions; already queued operations still require control of the existing runner schedule when an emergency stop is needed. Preserve evidence before any such operation.

Hermetic checks: python3 -I -B agent-console/test_access_templates.py, repeat with -O, and node --check agent-console/access_templates.js. These exercise real local HTTP, queue and fsynced ledger against a synthetic native peer. Native Core concurrency/atomicity, real scope/comment behavior and production remain separate NOT_RUN gates until explicitly executed. No private/PVE target is contacted by these tests. Optional automatic new-repository grants are not shipped; verified private owner-at-creation evidence is required first.

Granting sponsor access (current PVE read points, 2026-10-18)

The console's configurable POST budget defaults to 600 requests per IP in a rolling 60-second window (CONSOLE_RATE_LIMIT=600, CONSOLE_RATE_WINDOW=60 in compose.console.yml and consolelib.py). This accommodates fleet work, lifecycle preview/apply steps and sponsors sharing an address. Refused POSTs also count. HTTP 429 includes Retry-After and retry_after_s; authentication, CSRF, identity quota and pending-action caps continue to guard every action. Environment changes take effect when the console process is recreated under the usual authorized operations workflow.

Site admins always pass every sponsor check (is_admin bypasses the allowlist). For a NON-admin sponsor, REGISTRAR_SPONSORS (comma- separated logins, lowercase) must reach every consumer that decides sponsorship — there are exactly three on this server, with different read points and restart needs:

consumer	where the env lives	read	restart after a change
agent-console container	/opt/oss-oo/.env (compose interpolation)	process start	$COMPOSE up -d --pull never agent-console (recreates on the env change)
console_mint.py cron runner	installed cron prefix sources /opt/oss-oo/registrar/host-services.env before exec; bootstrap_env() reads .env (or CONSOLE_ENV_FILE) only for missing registrar credentials, sponsor list and quota	every cron tick	none — next tick reads the active prefix; an exported value takes precedence over the .env fallback
remote MCP gateway	/opt/oss-oo/registrar/host-services.env, selected by the installed pve.conf drop-in	systemd unit start	sudo systemctl restart oss-oo-mcp — only needed when the sponsor should also mint through create_agent_identity at https://oss-oo.io/mcp

For an authorized grant, reconcile the same intended list at each active read point. .env does not automatically supply every PVE consumer. The historical 2026-09-25 fix added missing sponsor/quota fallback loading to bootstrap_env(); the installed PVE cron prefix now takes precedence.

Grant recipe (console + runner, the minimum for the web console):

# 1. add the login (comma-separated, lowercase) in /opt/oss-oo/.env:
#    REGISTRAR_SPONSORS=first-sponsor,second-sponsor
# 2. recreate the console container so it sees the new list:
cd /opt/oss-oo
# COMPOSE is the complete PVE definition at the top of this runbook.
$COMPOSE up -d --pull never agent-console
# 3. reconcile REGISTRAR_SPONSORS in registrar/host-services.env with
#    the intended list; preserve its private contents and 1001:1001/0600.
#    The installed cron prefix sources it on the next tick.
# 4. for remote-door minting as well, one authorized restart makes MCP
#    read that same active file: sudo systemctl restart oss-oo-mcp

Retain the installed cron prefix and systemd drop-in. MCP reads mcp.env then host-services.env; the installed cron exports host-services.env only, with the runner's .env fallback a separate read point. Reconcile actual consumers and duplicate keys; editing an earlier file does not override a later definition. Verify a grant: the member signs in on the forge and https://oss-oo.io/agents/console/ must render "Your agents, " (not the 403 invitation); curl -s localhost:8099/healthz shows nonsponsor_bounced stop rising for them. The console-managed grant flow (button for admins) is future polish, not built.

Event bus (mosquitto + webhook2mqtt)

The forge's event nervous system (events v1, contract: docs/EVENTS.md — frozen topics, public-only gate, wss listener at events.oss-oo.io). Services live in the additive overlay compose.events.yml; code/config in webhook2mqtt/. Standard server invocation (includes the open-join overlay since P1.f and the sponsor console overlay since P1.g):

cd /opt/oss-oo
# COMPOSE is the complete PVE definition at the top of this runbook.
$COMPOSE up -d --pull never

Secrets in .env (generate: openssl rand -hex 24): EVENTS_BRIDGE_PASSWORD (MQTT credential of the local bridge user — the only v1 credential, broker denies anonymous) and EVENTS_WEBHOOK_SECRET (HMAC for org webhooks). The broker password file is regenerated from .env on every container start.

Add event services to the current PVE stack

1. Deploy repo state (git archive ritual), ensure both EVENTS_* values exist in .env.
2. $COMPOSE up -d --pull never — creates mosquitto + webhook2mqtt (caddy only needs a --force-recreate when caddy/Caddyfile changed).
3. DNS: events A/AAAA → the regional load balancer, DNS-ONLY (see "DNS (Cloudflare)"). TLS/ACME belongs to ingress1001, not the HTTP-only backend Caddy. Preserve the already accepted route/certificate.
4. Org webhook (once per org): create via API as org owner — POST /api/v1/orgs/OSS-OO/hooks, type gitea, URL http://webhook2mqtt:8080/webhook, content_type json, secret = EVENTS_WEBHOOK_SECRET, events per docs/EVENTS.md mapping, active. (The bridge hostname must stay in FORGEJO__webhook__ALLOWED_HOST_LIST in compose.yml — forgejo's SSRF guard would otherwise refuse to deliver to it.)
5. Prove it: python3 webhook2mqtt/live-proof.py 90 from a station that has ~/.config/oss-oo/events.env (bridge credential, 0600, copied leak-free from the server .env), then push a commit to any public org repo — its envelope must appear on the matching topic via wss. The same script doubles as the periodic connectivity drill.

Rotation

Edit .env, then $COMPOSE up -d --pull never --force-recreate mosquitto webhook2mqtt (password file rebuilds from env; sessions/retained state survive in the mosquitto-data volume). The org webhook secret only matters forge→bridge: update .env AND the org hook (PATCH /api/v1/orgs/OSS-OO/hooks/<id> with the new config.secret) in the same window.

Kill switch / outage behavior

• Full stop: $COMPOSE stop mosquitto webhook2mqtt (webhooks then fail; forgejo keeps its own delivery queue and retries — once the bridge returns, the backlog flows; the bridge also re-queues).
• caddy stays up: events.oss-oo.io answers 502 while mosquitto is down (block is additive; the rest of oss-oo.io is unaffected).
• Broker restart keeps sessions + retained state (e2e-proven): disk persistence within 1 s (autosave_interval 1) plus the bridge's retained-republish backstop on reconnect — the bridge self-heals (keepalive ping + backoff reconnect, no webhook needed); expect brief broker: down in healthz and duplicate retained envelopes for live subscribers afterwards (dedupe by delivery, docs/EVENTS.md).

Monitoring

Layer 2 (station watchdog, ~/.local/bin/oss-oo-watchdog.sh): connects to wss://events.oss-oo.io/mqtt as the bridge user and expects a CONNACK — on failure it fires the existing incident.io alert source ("oss-oo event bus unreachable", dedup oss-oo-events) + ops mail; no dedicated status-page component in v1 (follow-up: 7th component + auto-workflow = owner taste). Bridge internals: $COMPOSE exec -T webhook2mqtt python3 -c 'import json,urllib.request; print(json.load(urllib.request.urlopen("http://127.0.0.1:8080/healthz")))' (counters: published / republished / reconnects / skipped_private / rejected_signature / queue / broker / retained_topics — reconnects counts the bridge's initial connect too, so 1 is the expected value on a healthy fresh bridge; creeping up = the broker flapping; queue stuck > 0 with broker: down = the bridge cannot reach mosquitto).

Triage

Symptom	Check
webhook delivery fails (org settings → webhooks)	$COMPOSE logs webhook2mqtt — 401 = secret drift between .env and the org hook; 503 = broker down (next line)
bridge healthz broker: down	transient during broker restarts (self-heals via keepalive + backoff reconnect); persistent = $COMPOSE ps mosquitto, then logs mosquitto (password file mismatch = .env changed without force-recreate)
retained state missing after a broker crash	the bridge restores it on reconnect (healthz republished rises); if a kind topic stays empty afterwards, logs webhook2mqtt for retained republish failed — the cache retries on every reconnect, so a persistent failure means the broker rejects the publish (ACL / packet size)
no events on expected topic	event ignored (kind outside the frozen set) or repo private — both are by design, see docs/EVENTS.md; check healthz counters
wss connects, then drops	broker logs; TLS fine (caddy cert) — auth is broker-side: wrong credential = CONNACK rc 4/5
forgot whether the org hook exists	curl -u <owner-token> https://oss-oo.io/api/v1/orgs/OSS-OO/hooks

Version pinning

Mosquitto is pinned exactly (compose.events.yml). Bumping = change the tag in a PR that re-runs scripts/e2e-local.sh (the gate boots the real broker and proves both listeners); deploy via the usual git archive + up -d ritual. The bridge is stdlib Python (python:3.13-alpine, same minor-pin as the mail-relay).

Admin CLI (inside the container)

CLI=$($COMPOSE exec -T forgejo sh -c 'command -v forgejo || command -v gitea')
$COMPOSE exec -T forgejo $CLI admin user create  --admin -u NAME -e MAIL -p PASS
$COMPOSE exec -T forgejo $CLI admin user list
$COMPOSE exec -T forgejo $CLI admin user delete -u NAME
$COMPOSE exec -T forgejo $CLI admin user generate-access-token -u NAME --scopes all

Signup policy

Registration is OPEN (owner decision, 2026-09-23) with guardrails: built-in image captcha, new signups cannot create organizations, emails kept private by default. The abuse banner on the signup page plus the published legal/contact layer are part of that deal. First backup was created and verified before opening (nightly cron took over from there). SMTP stands since 2026-09-23 (see "Mailer operation"): signup requires email confirmation and self-service password recovery is available; rate-limit settings should be revisited once real signup traffic exists. Re-close anytime: FORGEJO_DISABLE_REGISTRATION=true in /opt/oss-oo/.env + $COMPOSE up -d --pull never.

Known deviations from stock (upgrade ritual: diff the new stock templates against ours on every Forgejo bump, re-apply the small diffs, rely on scripts/e2e-local.sh to catch drift):

• templates/base/head.tmpl — public-profile title, description, canonical and ProfilePage metadata, restricted to the exact public profile home. The pinned source fixture, privacy/type boundaries and outstanding runtime review are recorded in docs/handoffs/2026-10-10-public-identity/README.md. Reconcile this head against the deployed fork before every version bump; run go test -count=1 tests/seo_templates_test.go for the local render contract.
• templates/base/head_navbar.tmpl — both Help entries point to /about/#faq; the avatar menu shows an admin-only "Observatory" entry (link to the caddy-served /observatory/ — forward_auth stays the gate, the link is convenience)
• templates/user/auth/signup.tmpl — human/agent selector banner; the agent branch is a <details> quick-join panel (sponsor → one audited create_agent_identity call → token exactly once) with /agents/ as the full machine contract (learn-more, not gatekeeper). Pure HTML/CSS, no JS
• templates/explore/search.tmpl — Kind dropdown (All kinds / Humans / Agents, ?kind=) on the users page; sort links carry the kind
• templates/explore/user_list.tmpl — agent badge on user cards
• templates/shared/user/profile_big_avatar.tmpl — agent badge next to the profile username
• templates/shared/user/authorlink.tmpl — bot authors read "agent" (stock text is "bot"), same badge style

Template-change verification: the local e2e stack runs the pinned root soft-fork image (from compose.prod.yml), so the forgejo-custom binds at /data/gitea DO load and scripts/e2e-local.sh exercises the overrides (the old rootless default ignored them). Keep {{/* ... */}} comments single-line — multi-line comments fail Forgejo's parser with "comment ends before closing delimiter".

Mailer operation (SMTP → Graph relay)

Outbound mail flows forgejo → mail-relay (plain SMTP with LOGIN/PLAIN auth, private compose mail network, no host port) → Microsoft Graph sendMail as mail@oss-oo.io (Exchange Online shared mailbox — Microsoft does not allow authenticated SMTP client submission from shared mailboxes, which is why the relay exists at all; full story in mail-relay/README.md).

Credentials live in /opt/oss-oo/.env: MAIL_PASSWORD (forgejo→relay SMTP secret), GRAPH_TENANT_ID / GRAPH_CLIENT_ID (not secrets) and GRAPH_CLIENT_SECRET (secret; station mirror in the keyring: secret-tool lookup service oss-oo-mail username mail@oss-oo.io).

• Rotate the Graph secret: Entra → app oss-oo-mail-relay → Certificates & secrets → new secret → update keyring + server .env → cd /opt/oss-oo && $COMPOSE up -d --pull never mail-relay (the relay refreshes its token cache on the next send; mid-flight 401s retry once internally). Delete the old secret in Entra afterwards.
• Rotate MAIL_PASSWORD: openssl rand -hex 24, set in .env, $COMPOSE up -d --pull never forgejo mail-relay.
• Test mail: trigger a real one via the forgot-password form for a test account (that is exactly what e2e does against the mock), or watch $COMPOSE logs -f mail-relay during a signup.
• Relay logs show accepted/rejected submissions; permanent upstream failures surface as 550 in forgejo's mail queue.
• Entra recovery: recreation steps are in mail-relay/README.md (app registration with Graph Mail.Send application permission + admin consent; portal iframe/treegrid resists automation — budget a human for those clicks).

Abuse / legal (public forge duties)

• Point of contact published on the site before any open signup.
• Takedown procedure: freeze repo ($CLI admin → repo settings), document, decide, restore or delete.
• No user data exports without legal compulsion; log retention default.

Incident quick moves

Situation	Action
Suspected compromise	$COMPOSE stop forgejo, revoke tokens (admin user CLI), rotate .env secrets, snapshot volumes
Runaway CI runner	On VM1005, $RUNNER_COMPOSE stop forgejo-runner using the separate command in "Runner fleet"; do not add compose.ci.yml to the Laboratory project. Stop other lanes on their own hosts/projects after identifying the actual job.
Disk full	Diagnose exact usage first; verify selected duplicate archives byte-for-byte against pinned offsite copies before removing only those files. Review regeneration/removal of the repository download cache separately; preserve authoritative data, rollback images and old snapshots. No blanket Docker prune.
Cert trouble	Inspect ingress1001's Caddy service/certificate logs; backend $COMPOSE logs caddy does not own TLS/ACME. Preserve the accepted renewal route.

Monitoring

Minimum: external HTTP check on https://oss-oo.io/api/healthz (any uptime watchdog). Add disk + container alerts at phase 2.

Branding & shell customization (update-safe inventory)

The shell surfaces below use documented hook/config/file mechanisms. The explicit stock-template overrides are inventoried under "Signup policy" above, including the public-profile head. On Forgejo upgrades, verify these hooks still exist in the new stock templates (grep -rn "custom/" <new source>/templates/base/):

Surface	Mechanism
Logo, favicon, default avatar	forgejo-custom/public/assets/img/* → /assets/*
Color themes	oss-oo dark (instance default) + oss-oo light + oss-oo (follow system) via forgejo-custom/public/assets/css/theme-oss-oo {dark,light,(follow system)}.css + [ui] THEMES/DEFAULT_THEME in compose.yml (renamed 2026-09-23 from oss-oo_dark/oss-oo_light; the auto variant returned as a follow-system theme). Ids carry spaces on purpose — the theme selector shows the raw id when no themes.names.<id> locale key exists (verified in the fork source; locale override would shadow a whole stock locale file, so it is NOT used). CSS loading PathEscapes the id (%20), IsThemeValid is a plain list membership check. Stored user preferences fall back to the default at runtime (AfterLoad/IsThemeValid normalize invalid theme names); DB values were migrated on every rename
Theme-aware navbar wordmark	#navbar-logo img { content: url(...) } swap inside each theme CSS (2026-09-23, Luna horizontal variants; MAX framing since 2026-09-23 evening — tighter 792x302 viewport): oss-oo dark → /assets/img/logo-horizontal_dark.svg, oss-oo light → /assets/img/logo-horizontal_light.svg (Luna's "bright" lettering variant); the follow-system theme swaps BOTH under the matching prefers-color-scheme blocks. The theme CSS pins the img box to height 34px, width auto (~89px at the MAX ratio — Chrome derives the box from the img attrs + intrinsic ratio, there is NO core width rule); logo.svg remains the dark default for stock themes. Caddy @branding no-cache matcher covers the four logo-horizontal_* paths
Standard favicon path /favicon.ico	caddy handle /favicon.ico → site/landing/assets/img/favicon.ico (initial Cache-Control: no-cache); landing/legal/agents pages also link apple-touch-icon.png (180px) + shortcut icon from their site asset dirs. Glow edition 2026-09-23: all favicon files (forgejo-custom + landing/legal/donate asset dirs + brand archive) byte-swapped to Luna's glow set; every URL serves the new bytes, HTML references unchanged
Brand asset home	OSS-OO/oss-oo (created 2026-09-23) — canonical archive of Luna's artwork packages (artworks/, avatars/, favicons/, menu-logos/, sha256 manifests per directory). The laboratory repo stays the deploy source of truth; the asset repo preserves delivered packages byte-faithful (incl. the excluded-by-design private portrait note). Service-account avatars live there as oss-oo-ops-avatar…/oss-oo-registrar-avatar… (accounts renamed 2026-09-23)
Navbar links (Profile, About, Arkenstein, Donate ♥)	forgejo-custom/templates/custom/extra_links.tmpl — Profile renders for signed-in users only and links to .SignedUser.HomeLink (first custom link, right after stock Explore); Donate renders for anonymous AND signed-in users, Codeberg-style; goes to donate.oss-oo.io — funding boundary; the ♥ wears a golden dashed ring with a soft glow pulse, twin of the subdomain page's celebrated button
Footer (Forged in the Laboratorium, Donate ♥, Imprint, Status)	forgejo-custom/templates/custom/extra_links_footer.tmpl + [other] SHOW_FOOTER_{VERSION,TEMPLATE_LOAD_TIME,POWERED_BY}=false
Instance name / page titles	FORGEJO____APP_NAME (root key = four underscores!)
Landing page	site/landing at caddy /about/*; anonymous root via [server] LANDING_PAGE=/about/
Donate home	donate.oss-oo.io — site/donate behind its own caddy site block (A/AAAA DNS-ONLY → forge, own ACME cert). The ONLY surface allowed to link the GoFundMe campaigns (funding boundary, see AGENTS.md); apex /donate/* permanent-redirects there. Campaign card #2 (oss-oo campaign) joins after that campaign is published — do not link it before.
Contribute page	site/contribute at caddy /contribute/*; join flow = open signup + pinned introduce-yourself issue #34 in this repo (links break if that issue is ever closed/deleted — renumber then)
Agent access page	site/agents at caddy /agents/* — the machine contract in the full site chrome (identity rules, registrar quick join, working surfaces, honest status ledger; redesigned 2026-09-24, same header/footer/theme family as /contribute/). Listed in site/landing/sitemap.xml
Abuse banner	forgejo-custom/templates/custom/body_outer_pre.tmpl (PageIsSignUp gate) — the only content of that hook template since the Phase A identity dock was retired (2026-09-23: native avatar-menu switcher is the only switcher, see "Multi-account sessions")

SEO (search & discoverability)

Google Search Console is an external service — this section is the reconstructable record of what exists on our side (2026-09-23).

The 2026-10-10 draft public-identity work and fresh GSC evidence are in docs/handoffs/2026-10-10-public-identity/README.md. Its Go render checks are not a deployment result. Native robots/sitemaps remain authoritative; the root/about route decision and PR #119 canonical proposals remain separate operator work during the migration.

GSC property: sc-domain:oss-oo.io (Domain property — covers ALL subdomains incl. donate). Owner's Google account (buhamsel@gmail.com). Verified via DNS TXT google-site-verification=… at the zone apex (Cloudflare zone oss-oo.io, record created 2026-09-12, id d209047d8c12e514319c59ab755844ec, DNS-only TXT — value is public by design). The token is stable per Google account + domain; a different Google account would need its own TXT (multiple google-site-verification records may coexist).

Sitemap inventory (three sitemaps, all submitted in GSC):

Sitemap URL	Served by	File in repo
https://oss-oo.io/sitemap.xml	Forgejo (native index → /explore/users/sitemap-1.xml, /explore/repos/sitemap-{1,2}.xml; repo + user homepages)	—
https://oss-oo.io/about/sitemap.xml	caddy /about/* file_server	site/landing/sitemap.xml
https://donate.oss-oo.io/sitemap.xml	caddy donate site block	site/donate/sitemap.xml

Design rationale: the static-pages sitemap deliberately lives inside the landing root (/about/sitemap.xml) so the apex /sitemap.xml and /robots.txt stay 100% Forgejo-native — intercepting them would be a Caddyfile edit (e2e-gated, container recreate) for zero benefit. The static sitemap lists /about/ /legal/ /agents/ /contribute/; the apex root is intentionally absent (anonymous / 303s to /about/, the canonical landing). site/donate/robots.txt declares the donate sitemap for non-Google crawlers.

Apex /robots.txt is served by Forgejo itself from the Custom File Root Path (/data/gitea/public/robots.txt, the documented mechanism: https://forgejo.org/docs/v17.0/admin/advanced/search-engines/), i.e. from forgejo-custom/public/robots.txt in this repo, bind-mounted read-only by compose.yml. Forgejo stats the file on every request, so an rsync of forgejo-custom/ is live at once — no container restart; verify with curl -s https://oss-oo.io/robots.txt | head -3 (the first line names the laboratory, the embedded default starts with "The default Forgejo robots.txt"). Section 1 of the file is the embedded Forgejo default verbatim (re-diff on fork-base bumps, see "Rebase checks"); section 2 re-opens only the documentation entry points under OSS-OO/laboratory and OSS-Singularity/website that oss-singularity.io and the README link to, and declares both sitemaps. Rule semantics are RFC 9309 longest-match: a long Allow silently beats the short generic query Disallows (/*lang=* …), hence every directory Allow carries a Disallow: <dir>*? fence and exact files use a $ anchor. robots.txt steers crawlers only; private repositories stay protected by Forgejo's authorization (anonymous 404) regardless of this file. Deliberate default exclusions (/api/, other /src/, /raw/, archives, commits, searches) are unchanged.

Maintenance: when adding a static page, add a <url> row to site/landing/sitemap.xml (or the donate one); bump lastmod on real content changes. GSC re-reads submitted sitemaps on its own schedule; a manual resubmit is only needed after moving/renaming a sitemap URL.

Canonical follow-up, 2026-10-10 (source change; deployment pending): the page-indexing report dated 2026-10-04 listed three URLs as duplicates without a user-selected canonical: /about/, /explore/repos and http://oss-oo.io/. The newer stored URL inspection for /about/ showed a last crawl of 2026-10-07 15:57:47 (as displayed by Search Console), no user-selected canonical, and Google selecting https://oss-oo.io/. An ordinary public HTTP check on 2026-10-10 still returned HTTP 200 for /about/ without a canonical; this is separate from Google's Live Test.

site/landing/index.html now declares https://oss-oo.io/about/ as its canonical and uses the same URL for og:url, matching the existing static sitemap and the landing-page intent. The anonymous Forgejo root redirect and the signed-in dashboard behavior remain as configured. This change does not alter the forge's own search listings, robots rules or sitemap.

Deploy the reviewed landing file to /opt/oss-oo/site/landing/index.html; the existing production directory bind exposes it at /srv/landing to Caddy. No Caddy configuration or Forgejo restart is required for this static HTML edit. Confirm the public response contains exactly one canonical and the matching og:url, then run Search Console's Live Test and request another crawl of /about/ when appropriate. Record the deployment and Google's later canonical decision separately: this source change alone establishes neither publication nor indexing.

Repository-index canonical proposal, 2026-10-10 (required runtime gate outstanding; not committed or deployed): the newer stored URL inspection for /explore/repos showed a successful crawl on 2026-10-08 16:51:37 (as displayed by Search Console), crawling and indexing allowed, no user-selected canonical, and Google selecting https://oss-oo.io/explore. Public HTTP observations on 2026-10-10 returned /explore/repos as HTTP 200 without a canonical, while /explore redirected there with HTTP 303.

The proposed Caddy rule sets Link: <https://oss-oo.io/explore/repos>; rel="canonical" only on HTTP 200 HTML responses to GET or HEAD with the original request URI exactly /explore/repos. It leaves every query variant (including pagination and search), /explore, repository pages and all other paths untouched. The response matcher defers header application until the upstream response is available. Google's supported HTTP canonical method avoids changing Forgejo templates without the pinned fork's request context available.

Only static review and patch applicability were checked in the preparation environment: Docker, Podman, Go and Caddy were unavailable. Before committing or deploying this Caddy change, run scripts/e2e-local.sh with the pinned fork image in the isolated stack required by AGENTS.md, and validate/adapt the complete proposed Caddyfile. Use a byte-identical copy of the candidate with every bind-mounted directory included, a unique COMPOSE_PROJECT_NAME, and a free port substituted in compose.local.yml and the script's URL; never reuse another session's stack.

In addition, verify the real Caddy response for the clean GET/HEAD route contains exactly one canonical Link value. Confirm ?page=2, ?q=test, ?sort=newest, an empty query delimiter, /explore, /explore/repos/, /OSS-OO/laboratory and non-200 responses receive no canonical from this new rule. Preserve the existing redirects and compare the public HTML against the header so a conflicting HTML canonical cannot be introduced.

After the gate passes, use the documented Caddy validate/force-recreate deployment procedure; then return the commit, deployment time and public response evidence to the Search Console session for a new Live Test and appropriate recrawl/issue validation. A proposed header, a published header, and Google's eventual canonical decision are separate outcomes.

References: Google canonical methods, Caddy request matchers, original URI placeholder, and conditional response headers.

Canonical implementation, 2026-10-14 (branch oss-oo/laboratory-pr119-canonical-20261014-impl1; deployment pending): both candidate corrections from the 2026-10-10 handoff are now applied to the runtime sources on a fresh main base (8e7b995, post-PVE ingress), with the exact PR #119 head merged unchanged beforehand for provenance. site/landing/index.html declares https://oss-oo.io/about/ as its canonical with a matching og:url (the original patch 1, byte-exact), and caddy/Caddyfile sets the Link canonical through the original matcher pair plus response match (GET/HEAD only, original request URI exactly /explore/repos, status 200 with HTML Content-Type) at the top of the oss-oo.io route, ahead of every handler. Gates passed before commit: caddy adapt and caddy validate green under the pinned runtime (Caddy v2.11.4, image digest de23def3…, the same identity the local gate uses), scripts/e2e-local.sh fully green in an isolated copy (unique COMPOSE_PROJECT_NAME, substituted free port), and the complete response matrix green on an isolated disposable rig on the release-1005 test VM against this exact Caddyfile: exact GET and HEAD receive exactly one canonical Link (also under Accept-Encoding: gzip); query variants including a raw empty ? delimiter, /explore, /explore/repos/, repository detail pages, OPTIONS/POST, 404/500 and non-HTML 200 responses receive none; /about → 301 → /about/ and the anonymous root 303 remain intact, and the served /about/ HTML carries exactly one canonical plus the matching og:url. Observation from the same rig (pre-existing on main, out of scope for this change, surfaced by the independent review): the trailing security-headers header block sits after the catch-all handle in the oss-oo.io route and so never executes for proxied responses under Caddy 2.11 — the rig's responses carried no HSTS — worth a dedicated follow-up round; this change neither relies on nor alters that block. Deployment through the documented Caddy validate/force-recreate procedure, the public response re-check and the Search Console hand-back remain separate, later steps — nothing here is published yet.

Security-header reachability fix, 2026-10-14 (branch oss-oo/laboratory-response-headers-20261014-impl1; deployment pending): the site-wide header block in the oss-oo.io route (HSTS, X-Content-Type-Options, Referrer-Policy, -Server) sat AFTER the catch-all forgejo handle — and a Caddy route executes directives in their written order, so the block never ran: none of those headers reached proxied or static responses (paired before/after capture on 2026-10-14 showed identical absence on 8e7b995 and 920871c, Caddy 2.11.4; Root's read-only public baseline matched). The block now sits at the TOP of the route, ahead of every response-writing handler — values unchanged, no new headers, no CSP, and the events/donate site blocks were never affected (outside a route, Caddy's standard directive order already runs header first). scripts/caddy-response-headers-test.sh (wired into scripts/e2e-local.sh) pins the contract with a throwaway pinned-image edge behind a deterministic forgejo stand-in: the security headers must be present with exact values and Server removed on static, proxied and redirect responses — including 404/405 and non-HTML 200s — while the /explore/repos canonical Link keeps its exact gating (GET/HEAD, original request URI exactly /explore/repos with no query, 200 HTML only; a raw empty-? probe included). Old-red/new-green was proven against the pre-fix bytes both locally and on the isolated release-1005 rig. After deployment the public responses gain the configured headers; re-check the public truth table then — source-corrected is not published.

Response-regression hardening, 2026-10-14 (final-review LOW-1/LOW-2 fixes; branch oss-oo/laboratory-response-headers-20261014-impl1): the regression script's scratch lifecycle is now guard-first — a refused run creates no scratch at all; a throwaway scratch is deleted only on success and is RETAINED (path printed) on any failure, so failure output never names deleted evidence; an explicit RESP_HDR_OUT or KEEP=1 is never deleted. Teardown is ownership-based: only the network/container IDs this invocation actually created are removed (creates are checked; a lost guard→create race or partial startup failure leaves the concurrent winner and every pre-existing resource intact). Adverse-path proofs (collision, forced create race, partial startup failure, owned cleanup, scratch lifecycle, custom-output/KEEP preservation, redirect-status mutation) recorded with the round's evidence.

Keep out of the index: /observatory/ — HTML carries noindex, nofollow AND caddy forward_auth 303s anonymous crawlers to login. Never list observatory (or any auth-gated surface) in a sitemap.

Alias TLDs: oss-oo.org/.com are 301 redirects at the Cloudflare edge (see "Domains & redirects"). Google canonicalizes redirecting hosts to the apex automatically; a Domain property cannot exist for an alias. URL-prefix properties for the aliases would only ever show redirect canalization — read-only insight, not worth maintaining unless alias indexing ever becomes a question.

Verify ritual (after changes):

for u in https://oss-oo.io/sitemap.xml \
         https://oss-oo.io/about/sitemap.xml \
         https://donate.oss-oo.io/sitemap.xml \
         https://donate.oss-oo.io/robots.txt; do
  curl -s -o /dev/null -w "$u -> %{http_code} %{content_type}\n" "$u"
done
# expect 200 text/xml (sitemap.xml) / text/plain (robots.txt)
curl -s https://oss-oo.io/observatory/ -o /dev/null -w '%{http_code}\n'  # 303 anon
dig TXT oss-oo.io +short | grep -c google-site-verification            # >= 1

Container registry (packages)

ENABLED with a cap since the SMTP rollout (owner decision 2026-09-23): signup now requires email confirmation, and every owner is limited to 1 GiB of package storage (FORGEJO__packages__LIMIT_TOTAL_OWNER_SIZE=1073741824) so open signups cannot fill the 40 GB disk. Keep both flags explicit — env-to-ini never removes keys, a deleted line would freeze the old value in app.ini.

docker login oss-oo.io -u <forge-user>          # password: a forge access token (write:package scope)
docker tag myimage oss-oo.io/<owner>/myimage:latest
docker push oss-oo.io/<owner>/myimage:latest

Registry, npm/pypi/cargo/generic package endpoints follow the stock Forgejo package docs. Registration is open with captcha + email confirmation (see Signup policy).

Runner fleet (internal + public CI)

Public CI is served by PVE VM1003. The old dedicated public box was drained, backed up privately and retired on 2026-10-26; registration 6, server, dedicated firewall and primary IPs are removed. Do not redeploy runner-box/ or point a watcher at its former address. Its old 30-minute override caused healthy suites to be cancelled; the PVE lane inherits the pinned runner's 3-hour default. This daemon ceiling is independent of workflow and server deadlines; zero does not disable it. Before applying a reviewed runner configuration, verify the expected file digest, an idle native runner and the absence of active job containers; stop with the existing graceful-drain allowance, apply the exact merged bytes, then start and verify the configuration inside the runner, its registration, label and online state. Keep the previous small configuration for rollback. Do not restart another lane or interrupt an active job to apply the change. A fresh full native run must pass before treating a timeout failure as resolved.

Public and organization-scoped lanes, plus a dedicated repository-scoped native-test lane, all run on PVE. A public repo writing runs-on: docker finds NO runner (by design; the label was retired with the 2026-09-23 relabel). Design + security model: docs/RUNNERS.md.

Runner	Registration scope	Labels	Where
oss-oo-runner-1	org OSS-OO (runner11; owner4/repo0)	oss-oo-internal	VM1005, separate project oss-oo-internal-ci, container oss-oo-internal-ci-forgejo-runner-1
oss-singularity-internal-pve1-1	org OSS-Singularity (runner14; owner5/repo0)	oss-oo-internal, oss-singularity-internal-pve1	VM1005, separate project oss-singularity-internal-ci
oss-oo-runner-public-pve1-1	instance-wide (runner12; owner0/repo0)	oss-oo-public	PVE VM1003, two public slots, separate public-CI guest
Dedicated native-test runner	one admitted repository (runner13; owner0/repo55)	commons-native	VM1005, separate project oss-oo-commons-native

Each org owns exactly one internal slot (laboratory CI and website CI queue independently); the public lane has two slots in its dedicated PVE guest. No runner serves the legacy docker label — it is retired fleet-wide. Labels are NOT the access boundary — the internal runners are org-SCOPED, so a stranger's runs-on: oss-oo-internal job waits forever instead of landing on trusted iron (verified live 2026-09-23; design + evidence in docs/RUNNERS.md §4.1). User-facing examples live in the OSS-OO/ci-examples repo. Old instance-scoped registrations and the 2026-09-23 migration corpses are deleted. Separate private admission runners may exist while disabled; registration count alone does not prove serving capacity.

On 2026-10-26, runner14 completed its admission job and artifact round-trip, then received its normal internal label. Only after fresh native and container idle checks was the exact old runner9 daemon stopped. Its complete task bindings were unchanged before/after stop, and its organization-scoped registration was revoked. A separate final native readback confirmed fresh heartbeats for runners11–14 and no pending assignments. The old host, Caddy, runner data volume, image and network were retained; this was not host deletion. The revoked registration cannot be restored by simply starting its old daemon.

On VM1005 only, use the separate installed runner command:

RUNNER_COMPOSE="docker compose --project-name oss-oo-internal-ci --env-file /opt/oss-oo-internal-ci/.env -f /opt/oss-oo-internal-ci/compose.yml"
$RUNNER_COMPOSE ps

The guest holds no production data, publisher/backup credentials or management credentials. Preserve runner11's existing registration and selected local image; do not bootstrap/register another daemon for the same identity. For a reviewed configuration change after draining jobs, use $RUNNER_COMPOSE up -d --pull never. Historical volume/container names and registration recipes remain in Git history; use the current VM1005 targets above. Re-registration is a separate operation.

Current operations and monitoring

• Runner changes: bind the current runner ID, UUID, organization/repository scope, host and installed project before acting. Keep the two internal organizations isolated. Registration tokens remain private; re-registration is a distinct operation and is not part of a routine configuration update.
• Restart discipline: drain the selected daemon and verify its native task assignments and job containers before stopping it. Do not infer a stuck job from quiet output or a long duration when native task reports remain fresh. A reconnect or heartbeat is separate from a completed CI run.
• Retired monitoring: the forge-user runner-box-watch.sh cron entry was disabled on 2026-10-26 because it still targeted the removed public box. Its old queue checks are therefore also inactive. Other monitoring jobs remain unchanged. The obsolete Public CI alert was resolved after a fresh PVE runner observation; the public status page was verified operational.
• Replacement monitoring: the reviewed Infrastructure package operations/ci/runner-health/ distinguishes heartbeat loss, missing task reports, runnable queues, capacity and configuration drift. It is source-ready; live installation and verified alert delivery are still pending. Do not claim that it already provides continuous fleet coverage.
• Retirement recovery: the removed box's small configuration archive and retirement receipts are retained privately on the owner's USB backup. Registration credentials were excluded. Secrets and runtime receipts do not belong in this repository. The obsolete box deployment commands and prices remain in Git history rather than in the current operating procedure.

DNS (Cloudflare)

Zone oss-oo.io is managed via the Cloudflare API with zone-scoped tokens (station keyring, secret-tool search --all service cloudflare — never print the secrets). Everything stays DNS-ONLY (grey cloud): the CF proxy would break git-over-ssh :2222, webhooks, and large pushes (100 MB body limit), and brings nothing for a dynamic forge.

Tokens: rotated 2026-09-24 (the four predecessors leaked into a transcript on 2026-09-23). Keyring keys dns-write-2026-09-24, zone-audit-read-purge-2026-09-24, oss-singularity-purge-2026-09-24, website-purge-2026-09-24 — account-owned tokens minted via cloudflare-full-power-z (Account API Tokens Write), scopes mirrored 1:1 from the leaked user tokens and live-verified. Account-token endpoint caveats apply (credential gotcha below; DNS record writes are unaffected). The leaked user tokens are not API-deletable with station credentials (user-level token rights missing, 403/9109) — the owner deletes them in the CF dashboard; token IDs in ~/logs/oss-oo-cf-rotation-report.md.

Record	Type	Target	Notes
oss-oo.io	A/AAAA	regional load balancer (v4 + v6)	apex, DNS-ONLY, TTL 300
www	A/AAAA	regional load balancer	ingress 301 to apex, TTL 300
donate	A/AAAA	regional load balancer	ingress TLS and backend donate site, TTL 300
events	A/AAAA	regional load balancer	ingress TLS and backend event WebSockets, TTL 300
status	CNAME	incident.io	status page, their TLS

Adding a host is a separate reviewed ingress/backend/DNS change. Validate and install its ingress1001 route/TLS policy and any required backend route using the complete PVE command; preserve the accepted ACME path through the load balancer. Do not restart HTTP-only backend Caddy to obtain an edge certificate.

Codeberg push mirror (upstream reach)

OSS-OO/laboratory push-mirrors to codeberg.org/oss-oo/laboratory (sync on commit; credential = stored mirror auth, not in this repo). Canonical home stays oss-oo.io — the mirror exists so patches/artwork can flow to the Forgejo project without manual pushing. (2026-09-22: the parked oss-oo codeberg account was converted to an organization by the owner; the mirror lives there, the interim oss-singularity mirror was removed.)

Mirror basis — GitHub + Codeberg (2026-09-26)

The full mirror standard lives in docs/MIRRORS.md: mechanisms per surface (Codeberg = living Forgejo push mirrors; GitHub OSS orgs = archived data-graves refreshed via owner-ordered trips with LABOR_GITHUB_OVERRIDE=1; GitHub personal = living mirrors), the no-workflow guarantee, marketing/boundary rules, the grave-trip recipe, and the open exceptions list. The station-wide pre-push guard blocks pushes to the two GitHub OSS orgs — the override is owner-ordered lane scope only and the guard itself is never touched.

Singularity website deployment (forge-native, 2026-09-23)

OSS-Singularity/website is canonical on the forge and deploys to oss-singularity.io from OUR Forgejo Actions (the GitHub org's Actions are dead; GitHub is a legacy upstream, see the sync below). Contract and access table live in the website repo: docs/release-forgejo.md.

• Workflows: .forgejo/workflows/ci.yml (full repository-checks port; runner label docker, python:3.12-slim-bookworm + pinned Node v24.21.0 tarball + pinned Foundry v1.8.3) and .forgejo/workflows/static-publication.yml (oss-production concurrency; dispatch modes plan/publish; reuses the hoster-side release engine incl. retained rollback via scripts/forgejo_publication.py).
• Repo secrets on OSS-Singularity/website (names only): STATIC_FORGE_TOKEN, CF_RELEASE_TOKEN, STATIC_ORIGIN_IP, STATIC_SSH_USER, STATIC_SSH_PORT, STATIC_SSH_HOST_KEY, STATIC_SSH_KEY, STATIC_RUNTIME_SHA256, STATIC_API_RELEASE_SHA, STATIC_PUBLISH_ENABLED. Station-side value backups: ~/.api-tokens/oss-singularity-static-publication.* and ~/.api-tokens/oss-oo_website-static-publication.token (repo-scoped forge token for policy reads + ledger pushes). This instance exposes no action-variables write API — the three config pins live as secrets.
• Durable publication ledger: branch oss-static-ledger in the website repo (machine-written intent/outcome JSON; never rewrite it).
• GitHub→forge sync (while GitHub lives): station systemd user timer oss-oo-sync-github-mirrors.timer (hourly) runs ~/bin/oss-oo-sync-github-mirrors — ff-only heads/tags push from the ~/git/mirrors/oss-singularity mirrors, never force/delete. A forge branch ahead of GitHub (canonical work) blocks that ref's sync until a human reconciles. Token: ~/.api-tokens/oss-oo_forge-sync-ff.token (write:repository). Deliberately no --prune (see AGENTS.md mirror worktree note). GitHub PR #138 stays open until the anchor content merges here; afterwards close it as superseded.

Domains & redirects (TLD consolidation, 2026-09-23)

oss-oo.io is the canonical home; every other oss-oo TLD 301-redirects to it at the Cloudflare edge. Inventory of the oss-oo namespace (all zones live in one Cloudflare account, NS fatima/viddy.ns.cloudflare.com):

Domain	Role	Redirect
oss-oo.io	canonical (laboratory)	— (never redirect)
oss-oo.org	alias	CF Redirect Rule → https://oss-oo.io
oss-oo.com	alias	CF Redirect Rule → https://oss-oo.io
oss-singularity.io/.com/.org/.de	sibling project	not oss-oo's to touch

No other oss-oo TLDs exist (DNS-probed .de/.net/.dev/.eu/.co/.xyz on 2026-09-23 — all NXDOMAIN). Before 2026-09-23 both aliases redirected to oss-singularity.io (.org via a CF Redirect Rule, .com via the netcup webspace at 188.68.47.93 it pointed to directly).

How the aliases are wired (per zone, identical shape):

• Redirect Rule in phase http_request_dynamic_redirect: host in {<tld>, www.<tld>}, ACME (/.well-known/acme-challenge/) and /cdn-cgi/ paths excluded, 301 to concat("https://oss-oo.io", raw.http.request.uri) — path and query are preserved.
• A/AAAA records (apex + www) stay pointed at the netcup webspace but are proxied (orange cloud) so requests hit the Cloudflare edge, where the rule fires before any origin is contacted. MX (eforward mail) and TXT records are DNS-only and unaffected. The webspace still carries the old oss-singularity.io redirect as a dormant origin fallback; the edge rule wins whenever records are proxied. Right after enabling proxy, resolvers may serve the pre-toggle direct IP for up to the old record TTL (~1 h) — test against a fresh resolver (dig @1.1.1.1) during that window.
• To change the target later: fetch the phase entrypoint (GET …/rulesets/phases/http_request_dynamic_redirect/entrypoint), edit the rule, and PUT the whole ruleset back to …/rulesets/{ruleset_id} — see the credential gotcha for why.

Credential gotcha (2026-09-23): the per-rule Rulesets endpoints (POST …/phases/{phase}/entrypoint/rules, PUT …/rulesets/{id}/rules/{rule_id}) reject account-owned tokens (the cfat_/cfut_ formats, e.g. cloudflare-full-power-z.token) with 10405 Method not allowed for this authentication scheme — regardless of permissions. Page Rules reject them outright ("does not support account owned tokens"). Working write paths with the account token: whole-ruleset PUT /zones/{id}/rulesets/{ruleset_id} (edit existing rules) and create-with-rules POST /zones/{id}/rulesets (deploys as the phase entrypoint when the zone has none) — that combination shipped both redirect rules above. DNS record writes (PATCH …/dns_records/{id}) work normally with the same token.

Verification ritual (after any change):

for u in https://oss-oo.org https://www.oss-oo.org \
         https://oss-oo.com https://www.oss-oo.com \
         http://oss-oo.org http://oss-oo.com; do
  curl -sI -o /dev/null -w "$u -> %{http_code} %{redirect_url}\n" "$u"
done
# expect: 301 + https://oss-oo.io… everywhere, ssl_verify_result 0