# RUNNERS — CI runner fleet: design & security concept Public CI = strangers run arbitrary code in containers on our iron. This document is the design baseline for growing the runner fleet step by step: what isolation exists today (measured, not assumed), what hardening the forgejo-runner offers, who may use public CI (owner decision), the abuse model, capacity math, and the scaling path. Implementation follows in phases, each behind an explicit owner GO — anything that costs money and any loosening of security is Claudiu's call, never a default. Historical planning status: **Step A = this document (design only, no live change).** Step B executed 2026-09-23 with an owner override: **public runners get dedicated hardware immediately** — no co-located public slots on the forge box, so public jobs can never resource-DoS oss-oo.io. As-built: §7. Step C (further scale-out / on-demand burst) stays behind owner GOs. ## Current serving fleet (verified 2026-10-26) - Public CI runs on PVE VM1003, registration 12 `oss-oo-runner-public-pve1-1`, with two slots and the `oss-oo-public` label. A complete native functional run finished there in 32m20s. - The two organization-scoped internal lanes run on PVE VM1005: registration 11 serves OSS-OO and registration 14 serves OSS-Singularity. They retain separate registrations, projects and job networks. - Registration 13 is a separate repository-scoped native-test runner on VM1005. Its `commons-native` label does not grant other repositories access. A complete native PostgreSQL/HTTP CI run and its artifact round-trip passed. - Registration 14 passed its own real job and artifact round-trip before receiving the normal `oss-oo-internal` label. The previous registration 9 was then stopped while idle and revoked, with no new or changed task assignments. Its host and non-runner services remain retained separately. - The old dedicated public box and registration 6 were retired on 2026-10-26 after a verified private backup and clean drain. The server, its dedicated firewall and both primary IPs have been removed. `runner-box/` is retained as historical source, not an active deployment target. - The old box-specific watcher must not run against the retired address. Its active VM1002 cron entry was disabled on 2026-10-26. Replacement source observes native task reports, queue eligibility and loaded runner policy; continuous activation and routed notification acceptance are still pending. The dated design and measurements below remain historical context. Use this section and the current fleet runbook for placement decisions. ## 1. Original state (verified 2026-09-23) - Exactly one runner: container `oss-oo-forgejo-runner-1` (`data.forgejo.org/forgejo/runner:13.2.0`), part of the `oss-oo` compose project via `compose.ci.yml` (docker.sock mounted, root). - No `config.yml` — the daemon runs on pure defaults with the registration file `/data/.runner` (id 2, name `oss-oo-runner-1`, address `https://oss-oo.io`, one label `docker:docker://node:24-bookworm`). Default-config consequences: - `runner.capacity: 1` → **single slot**; laboratory CI (~2 min) and website CI (~9–11 min) queue behind each other (~8 min latency is normal queuing, not the 502-limbo). - `runner.timeout: 3h`, `container.network: ""` (per-job ephemeral network), `container.privileged: false`, `container.valid_volumes: []`, `container.docker_host: "-"` (no docker socket in job containers), `container.force_pull: false`, `cache.enabled: true` (internal cache server, `$HOME/.cache/actcache`). - Actions are enabled instance-wide (`[actions] ENABLED = true`; per-repo/per-org toggles exist, see §5). Signups are open with captcha + email confirmation; `MAX_CREATION_LIMIT=-1`; registry capped at 1 GiB/owner. - Server: Hetzner CX23 (`oss-oo-forge-1`, FSN1, 2 vCPU / 4 GB / 40 GB, €5.49/mo net). Idle measurements 2026-09-23: containers ≈ 280 MiB (caddy 37, forgejo 113, db 102, runner 9, mail-relay 19), host total used ≈ 850 MiB, **available ≈ 2.9 GiB**, disk 24 % used (28 G free). Operational quirks (unchanged, see RUNBOOK): after forgejo restarts, tasks can park in a 502-report limbo → `docker restart oss-oo-forgejo-runner-1` releases them (runner is in the compose project, plain `docker restart` — never while a sibling job runs); deploys that include `compose.ci.yml` recreate the runner container and kill running CI ("context canceled"). ## 2. Isolation model — what job containers can actually reach The docker executor runs job containers as **siblings** of the runner (daemon reached via `/var/run/docker.sock`), not nested. With the default `container.network: ""` every job gets its own ephemeral bridge network. Measured 2026-09-23 with a probe container on a separate bridge network (the exact situation of a job container): | Target from a job-like container | Result | |---|---| | forgejo internal `:3000` (oss-oo_default) | **blocked** | | caddy internal `:443` (oss-oo_default) | **blocked** | | mail-relay `:25` (oss-oo_mail) | **blocked** | | postgres `:5432` (oss-oo_default) | **blocked** | | `oss-oo.io:443` public (via NAT) | reachable (needed to clone/publish) | | host gateway ports (e.g. git-ssh `:2222`) | reachable (public services) | Docker's inter-bridge isolation (DOCKER-ISOLATION iptables rules) walls job containers off from the internal compose networks — that is the load-bearing wall and it holds today. Egress to the internet is open (NAT): jobs can fetch dependencies, which also means they can mine, spam, or exfiltrate — see §6. Residual risks that no bridge isolation covers: - **The runner daemon is root with docker.sock** — the runner itself is trusted (pinned image, our registration); a runner compromise = host compromise. Job containers do NOT get the socket (`docker_host: "-"` default). - **Workflows can pick any container image** (`container: image:` in the workflow overrides the label image). Any image from any registry runs with the same sandboxing as label images. Mitigation = caps + timeout + egress posture (§6), not image pinning. - **Host ports are reachable** from job networks (2222/80/443) — these are public services anyway; keep it that way (never bind a private service to the host bridge gateway). ## 3. Hardening toolbox (forgejo-runner v13 + instance side) Runner config (`config.yml`, generated reference at `code.forgejo.org/forgejo/runner` — `internal/pkg/config/config.example.yaml`; the historical public-box configuration remains in `runner-box/config.yml`): | Option | Default | Hardened setting for a public runner | |---|---|---| | `runner.capacity` | 1 | 1 (one public slot; raise only with headroom) | | `runner.timeout` | 3h | inherit the pinned runner default, matching the PVE public lane | | `runner.shutdown_timeout` | 3h | 10m — bounded graceful drain on stop | | `runner.labels` | from `.runner` | `oss-oo-public:docker://docker.io/library/node:24-bookworm` (fully-qualified pin) | | `container.network` | "" (per-job) | keep "" (ephemeral per-job networks + isolation §2) or a dedicated `internal: true` network (§6 egress options) | | `container.privileged` | false | false | | `container.options` | — | `--memory=1g --cpus=0.75 --pids-limit=256 --security-opt no-new-privileges` (job-container resource caps) | | `container.valid_volumes` | [] | [] (no host mounts into public jobs) | | `container.docker_host` | "-" | "-" (never expose the socket) | | `container.force_pull` | false | true for the public runner (fresh image, no stale tag cache) | | `cache.enabled` | true | **false** for the public runner (cache poisoning is a documented Forgejo attack vector + disk growth) | The daemon ceiling also applies when a workflow requests more time. On 2026-10-26 two healthy functional suites were cancelled by the older box's 30-minute override; the same head completed on the PVE public lane in just over 32 minutes. The older box was aligned to the same **3-hour default** before its retirement. The PVE runner retains that pinned native default. This removes the inconsistent local override; it does not disable the native ceiling. In v13.2.0, setting `timeout: 0` or a negative duration also retains that default. Workflow and server deadlines are separate and may be lower. Check all effective deadlines when diagnosing a cancellation, and review them when upgrading runners. A successful whole job is required; partial passing output is not a substitute. A quiet log by itself is not evidence that a job or runner has stopped working. Instance side (app.ini / env, applies fleet-wide — our own repos too): - `[actions] LOG_RETENTION_DAYS` / `ARTIFACT_RETENTION_DAYS` (defaults 365 / 90): every public job stores logs and artifacts on the forge data volume for that long. For the public era: **30 / 14 days** is plenty for a free tier (tighten in the Step B PR; verify effective defaults at rollout). - Per-repo and per-org Actions unit toggles exist today (repo settings; org default units) — the moderation lever, see §5. - Upstream quota engine (disk soft-quota shipped; per-user CI build minutes in development) — nothing stable to build on yet; we are our own upstream and can adopt or patch later once it matures. ## 4. Who runs on what — label architecture and runner scoping Forgejo schedules a job to any runner whose labels match `runs-on:`; a runner serves exactly the labels it registered. **Labels alone are NOT an access boundary** — verified live 2026-09-23 (§4.1): a throwaway user's repo requesting `runs-on: oss-oo-internal` was dispatched to a trusted internal runner within one second. The load-bearing gate is **runner registration scope** (native Forgejo): a runner registered with an org-scoped registration token (`owner_id` set on the registration) only ever receives jobs from that org's repositories — regardless of which labels a stranger requests. Current fleet: - **Internal lane, org-scoped:** `oss-oo-runner-1` (registration 11) is registered to **OSS-OO**; `oss-singularity-internal-pve1-1` (registration 14) to **OSS-Singularity**. Both run in separate projects on PVE VM1005. Each org has its own slot — laboratory CI and website CI no longer queue behind each other. Label on both: `oss-oo-internal:docker://docker.io/library/node:24-bookworm`. Registration 14 also retains its unique admission label `oss-singularity-internal-pve1`; its organization scope is unchanged. - **Native-test lane, repository-scoped:** registration 13 serves only its admitted repository with `commons-native`. Its separate execution policy and successful native tests do not change either general internal lane. - **Public lane, instance-scoped:** `oss-oo-runner-public-pve1-1` on PVE VM1003 serves only `oss-oo-public`; every account's repo may request it. The older `oss-oo-runner-public-1` registration has been retired. - The legacy `docker` label is retired fleet-wide; a repo writing it finds no runner (by design). - GitHub-default labels (`ubuntu-latest`, `ubuntu-24.04`, …) never had a runner either: workflows requesting them wait forever and pile up zombie runs (22 in the zcode-dispatch repos, cancelled 2026-09-24). Forge-side workflows must request `oss-oo-public` / `oss-oo-internal`. The source at `2e0ea7cea712d468278d30f265c6a53d0305bba1` (`modules/actions/workflows.go`, `ListWorkflows`) selects the **first existing directory** in `.forgejo/workflows`, `.gitea/workflows`, `.github/workflows` order. It does not merge files by basename. Even an empty `.forgejo/workflows` directory suppresses later fallbacks; the adjacent upstream unit tests explicitly cover this behavior. Put every intended Laboratorium workflow in `.forgejo/workflows/`. Repositories keeping a GitHub mirror can retain its separate config, but a `.github` file without a same-named native file does not run here when the native directory exists. This corrects the previous per-basename claim; it does not establish the cause of historical runs. The source/version correspondence is not a deployed-image attestation. - User-facing docs for contributors: the **`OSS-OO/ci-examples`** repository (label usage, caps, fair-use rules, copy-paste workflows, proven green on the public lane). ### 4.1 Label-collision finding & fix (2026-09-23, release-hardening sweep) - **Verified:** push to a throwaway non-member repo with `runs-on: oss-oo-internal` → task 197 executed on `oss-oo-runner-internal-2` (trusted, uncapped default config) — the label string is requestable by anyone and labels were the only split. - **Fix (native, no fork patch):** both internal runners were re-registered with org-scoped tokens minted via `GET /api/v1/orgs/{org}/actions/runners/registration-token` (org owner). Dispatch-side scope filter confirmed in source (`GetAvailableJobsForRunner`: `owner_id != 0` restricts candidate jobs to that owner's repositories). - **Proof both directions:** post-fix, the same throwaway repo's internal job stayed `waiting` forever (no runner can take it) while its `oss-oo-public` control job ran green; real workloads proven too — website `ci.yml` re-run green on the OSS-Singularity-scoped runner, laboratory CI green on the OSS-OO-scoped runner. - **Watchdog:** `scripts/runner-box-watch.sh` alerts when a foreign-repo job requests `oss-oo-internal` (it will wait forever — that is now an abuse/accident signal, not a hole) or when one of OUR internal jobs waits >2 h (outage signal). - **Rollback anchors:** pre-swap `.runner` files are kept beside the live ones on each box (`/data/.runner.bak-instance-scope` in the runner volumes); restoring one + restarting the daemon reverts to instance scope (old registrations would need re-minting after 2026-09-23 row cleanup). ## 5. Who gets public CI — options for the owner Native capability facts (verified, not invented): Actions can be toggled per repo and per org default-units today; global on/off via `[actions] ENABLED`; per-user CI-minute quotas do NOT exist upstream yet (§3). Options, from most open to most controlled: 1. **All email-confirmed users, Actions ON by default.** Zero admin overhead, maximum welcome. Abuse surface = anyone with a confirmed mailbox; contained by §6 caps + label split + abuse response. Recommended start for the "public - free!" promise, paired with option 3's moderation lever in the back pocket. 2. **Per-repo opt-in (Actions OFF by default for new repos).** New repos start without Actions; users request the flip (issue to owners / future automation). Strongest moderation without quotas; costs a manual step and friction against the open-door ethos. 3. **Org members only / selected orgs.** Public CI as a privilege of the orgs we curate (OSS-OO, OSS-Singularity). Simple, but excludes exactly the outside contributors public CI is meant to serve. 4. **Hybrid (recommended):** option 1 at launch with §6 hardening + monitoring; if abuse appears, degrade to option 2 by flipping org default units (one setting) and keep trusted repos enabled. Rented capacity ("kaufbare" step, Step C) can layer a premium label (e.g. `oss-oo-public-fast`) later — same gating question returns then. ## 6. Abuse vectors & countermeasures | Vector | Countermeasure | |---|---| | Crypto mining (CPU) | `--cpus=0.75` cap per job, 20m timeout, capacity 1 (one abuser max), queue visible in forge admin; 2 vCPU host keeps forge responsive | | Memory exhaustion (OOM host) | `--memory=1g` per job; ~2.9 GiB available + 1 job max; `--pids-limit=256` (fork bombs) | | Disk fill (images, workspaces, logs, artifacts) | no cache (poisoning + growth), `force_pull`, log/artifact retention 30/14d, disk alarm at 70 % exists, `docker system prune` in RUNBOOK incident table; job workspaces live in the runner volume (observable, one `docker rm` away) | | Egress spam / exfil from our IP | THE open risk of increment 1 (egress is NAT-open): caps + timeout + label split + abuse response (takedown, runner kill: it runs in its OWN compose project → `docker stop` without touching forge). Harder walls = §6.1 options, owner picks | | Fork-PR untrusted code | Verified trust model: `pull_request` from forks runs the PR's workflow WITH NO repo secrets and a read-only automatic token (safe by design); `pull_request_target` runs base-branch workflows WITH secrets — our repos must never check out PR heads in `pull_request_target` workflows (policy note; grep our repos at Step B) | | Secret exposure on runners | No instance secrets in public-lane envs; runners carry only their registration secret (`.runner` file, volume-internal); public jobs get at most the read-only fork token | | Malicious job image (`container: image:`) | Inherent to the docker executor; contained by the same sandbox (§2) + caps + timeout; egress wall (if chosen) is the real answer; `force_pull` avoids stale-tag tricks | | Runaway queue / job storms | capacity 1 per public runner; queue depth visible in forge admin; monitoring hook (§8) mails on sustained backlog | ### 6.1 Egress posture options (owner decision for Step B) 1. **Open egress + caps (recommended increment 1):** jobs can fetch dependencies (npm, pip, apt) — the runner is actually useful. Risk: our IP's reputation (spam) and mining-pool egress; mitigated by CPU/time caps and instant kill switch. Free. 2. **Blocked egress (`internal: true` docker network for job containers):** jobs lose internet (only oss-oo.io via explicit exception is complex) — kills mining/spam utterly but also kills dep-fetching; public CI becomes near-useless for real projects. Free. Fallback if abuse hits before better tooling exists. 3. **Allowlist egress proxy (follow-up, Step C era):** dedicated runner box routes job egress through a filtering HTTP(S) proxy (package registries + forge only). The honest long-term answer; needs the dedicated box (isolation of failure domains) + config work. Costs the Step C box. ## 7. Capacity math & Step B recommendation (no new spend) CX23 budget: 3.8 GiB total, ~2.9 GiB available idle, keep ≥1 GiB headroom (spikes, page cache, backups) → **~1.9 GiB CI budget**. - Internal lane job peaks ≈ 1.2–1.5 GiB (website Foundry+Node build); laboratory CI ≈ 0.1 GiB. - Public lane job capped at 1 GiB (`--memory=1g`), worst case both lanes building simultaneously: ≈ 0.9 (stack) + 1.5 (internal) + 1.0 (public) = 3.4 GiB < 3.8 GiB — tight but safe, and both-full is rare; the 0.75 CPU cap leaves 1.25 vCPU for forge + internal lane. **Step B as built (2026-09-23, owner GO + hardware override):** the co-located variant below is superseded — public jobs run on a **dedicated box** so they can never contend with the forge stack: - **Box `oss-oo-runner-1`**: Hetzner **cx33** (4 vCPU / 8 GB / 80 GB, fsn1, €8.49/mo net — under the approved cpx21 frame because cpx21 is deprecated/unorderable in the EU since 2025-12; verified live at order time), docker-ce app image, user `runner`, Hetzner firewall (inbound SSH + ICMP only). Station alias `oss-oo-runner`, forge→box alias `oss-oo-runner-box` (see RUNBOOK). - **Runner `oss-oo-runner-public-1`**: compose project `oss-oo-runner` from this repo (`runner-box/`), full §3 hardening matrix (`runner-box/config.yml`), label `oss-oo-public` only, registered with a CLI-minted token that never left the server/box shells. - **Internal lane**: existing runner re-registered to `oss-oo-internal` (dual-label first so no CI broke mid-migration, `docker` label dropped after all our workflows moved). The singularity foundation box's runner (which had been sharing load under the legacy `docker` label with DEFAULT config next to the publication engine) was re-registered as `oss-oo-runner-internal-2` in the same window — the legacy label is extinct fleet-wide and the internal lane gained a second slot (forge + foundation box). Later the same day both internal runners were **org-scoped** (§4.1) — OSS-OO and OSS-Singularity each own exactly one internal slot; instance-scoped dead registrations removed. - **Instance side**: `[actions] LOG/ARTIFACT_RETENTION_DAYS = 30/14`. - **Monitoring**: `scripts/runner-box-watch.sh` on the forge (cron 15 min: box reachability, stuck jobs >1 h, disk ≥70 %) through the ops-mail channel; kill switches in the RUNBOOK fleet section. *Original (superseded) co-located design, kept for the record:* CX23 budget math — 3.8 GiB total, ~2.9 GiB available idle, ≥1 GiB headroom → ~1.9 GiB CI budget; internal job peaks ≈ 1.2–1.5 GiB; a public job capped at 1 GiB fit alongside (worst case ≈ 3.4 GiB). The owner override removes this coupling entirely; the numbers remain valid as the argument for WHY public jobs got their own iron. ## 8. Monitoring & operations (fleet section of the RUNBOOK, lands with Step B) - Queue/job pulse: small server-side cron (forge user, alert-mail channel) that counts running job containers and mails when the public slot is busy >N consecutive checks or a job approaches the timeout — reuses the existing ops-mail path, no new services. - Kill switches: public runner is one `docker stop` (own project); whole fleet = `[actions] ENABLED=false` + runner stops (forge stays up). Both documented next to the existing "Runaway CI runner" row. - Restart discipline (unchanged): runner restarts only with an idle Actions queue; the 502-limbo release procedure stays in the RUNBOOK. - Disk: 70 % alarm already live; public-lane disk budget = runner volume + images + 14d artifacts — covered by the same alarm. ## 9. Step C — scale-out (separate round, owner GO, costs money) Live fsn1 pricing via Hetzner API (net, 2026-09-23 — **cpx21 is deprecated in the EU** (unavailable since 2025-12-31, US locations only) and was replaced by cx33 at order time): | Type | vCPU / RAM / disk | €/mo net | €/h net | Fit | |---|---|---|---|---| | cx23 | 2 / 4 GB / 40 GB | 5.49 | 0.0088 | the forge box itself | | **cx33** | **4 / 8 GB / 80 GB** | **8.49** | 0.0136 | **the public runner box (as built, Step B)** | | cpx11 | 2 / 2 GB / 40 GB | 5.49 | 0.0088 | smallest burst box | | cax21 (ARM) | 4 / 8 GB / 80 GB | 10.49 | 0.0168 | best €/core, ARM image caveats | | cpx31 | 4 / 8 GB / 160 GB | 17.49* | 0.0280 | sustained load + egress-proxy (*stale quote — verify) | Two shapes, combinable: - **Dedicated always-on runner host** (recommended when public CI is routinely busy or before raising capacity/egress posture): cpx21 in FSN1, docker + forgejo-runner only, same §3 config with higher capacity (2–3), registered to oss-oo.io over HTTPS. Nothing stateful worth backing up (jobs are ephemeral; registration secret is re-mintable) — a rebuild script replaces backup. Monitoring: same cron pattern + the status-page "CI runners" component. - **On-demand burst box** (`scripts/runner-burst.sh`, future): create via `api.hetzner.cloud` with the station token → cloud-init installs docker + runner and registers via a freshly minted registration token → hourly billing caps cost (cpx21 ≈ €0.37/day at 24h; burst windows cost cents) → idle-shutdown (no job for N minutes → deregister + delete server). Cloud-init + firewall (SSH only from station/forge) documented before first use; registration tokens are secrets and stay server/box-side. Trigger for Step C: public queue latency hurts (jobs waiting >10 min routinely), sustained CPU saturation, or the wish for premium/rented labels ("kaufbare" capacity) — each its own owner GO.