laboratory/docs/RUNNERS.md

RUNNERS — CI runner fleet: design & security concept

Public CI = strangers run arbitrary code in containers on our iron. This document is the design baseline for growing the runner fleet step by step: what isolation exists today (measured, not assumed), what hardening the forgejo-runner offers, who may use public CI (owner decision), the abuse model, capacity math, and the scaling path. Implementation follows in phases, each behind an explicit owner GO — anything that costs money and any loosening of security is Claudiu's call, never a default.

Historical planning status: Step A = this document (design only, no live change). Step B executed 2026-09-23 with an owner override: public runners get dedicated hardware immediately — no co-located public slots on the forge box, so public jobs can never resource-DoS oss-oo.io. As-built: §7. Step C (further scale-out / on-demand burst) stays behind owner GOs.

Current serving fleet (verified 2026-10-26)

• Public CI runs on PVE VM1003, registration 12 oss-oo-runner-public-pve1-1, with two slots and the oss-oo-public label. A complete native functional run finished there in 32m20s.
• The two organization-scoped internal lanes run on PVE VM1005: registration 11 serves OSS-OO and registration 14 serves OSS-Singularity. They retain separate registrations, projects and job networks.
• Registration 13 is a separate repository-scoped native-test runner on VM1005. Its commons-native label does not grant other repositories access. A complete native PostgreSQL/HTTP CI run and its artifact round-trip passed.
• Registration 14 passed its own real job and artifact round-trip before receiving the normal oss-oo-internal label. The previous registration 9 was then stopped while idle and revoked, with no new or changed task assignments. Its host and non-runner services remain retained separately.
• The old dedicated public box and registration 6 were retired on 2026-10-26 after a verified private backup and clean drain. The server, its dedicated firewall and both primary IPs have been removed. runner-box/ is retained as historical source, not an active deployment target.
• The old box-specific watcher must not run against the retired address. Its active VM1002 cron entry was disabled on 2026-10-26. Replacement source observes native task reports, queue eligibility and loaded runner policy; continuous activation and routed notification acceptance are still pending.

The dated design and measurements below remain historical context. Use this section and the current fleet runbook for placement decisions.

1. Original state (verified 2026-09-23)

• Exactly one runner: container oss-oo-forgejo-runner-1 (data.forgejo.org/forgejo/runner:13.2.0), part of the oss-oo compose project via compose.ci.yml (docker.sock mounted, root).
• No config.yml — the daemon runs on pure defaults with the registration file /data/.runner (id 2, name oss-oo-runner-1, address https://oss-oo.io, one label docker:docker://node:24-bookworm). Default-config consequences:
  • runner.capacity: 1 → single slot; laboratory CI (~2 min) and website CI (~9–11 min) queue behind each other (~8 min latency is normal queuing, not the 502-limbo).
  • runner.timeout: 3h, container.network: "" (per-job ephemeral network), container.privileged: false, container.valid_volumes: [], container.docker_host: "-" (no docker socket in job containers), container.force_pull: false, cache.enabled: true (internal cache server, $HOME/.cache/actcache).
• Actions are enabled instance-wide ([actions] ENABLED = true; per-repo/per-org toggles exist, see §5). Signups are open with captcha
  • email confirmation; MAX_CREATION_LIMIT=-1; registry capped at 1 GiB/owner.
• Server: Hetzner CX23 (oss-oo-forge-1, FSN1, 2 vCPU / 4 GB / 40 GB, €5.49/mo net). Idle measurements 2026-09-23: containers ≈ 280 MiB (caddy 37, forgejo 113, db 102, runner 9, mail-relay 19), host total used ≈ 850 MiB, available ≈ 2.9 GiB, disk 24 % used (28 G free).

Operational quirks (unchanged, see RUNBOOK): after forgejo restarts, tasks can park in a 502-report limbo → docker restart oss-oo-forgejo-runner-1 releases them (runner is in the compose project, plain docker restart — never while a sibling job runs); deploys that include compose.ci.yml recreate the runner container and kill running CI ("context canceled").

2. Isolation model — what job containers can actually reach

The docker executor runs job containers as siblings of the runner (daemon reached via /var/run/docker.sock), not nested. With the default container.network: "" every job gets its own ephemeral bridge network. Measured 2026-09-23 with a probe container on a separate bridge network (the exact situation of a job container):

Target from a job-like container	Result
forgejo internal :3000 (oss-oo_default)	blocked
caddy internal :443 (oss-oo_default)	blocked
mail-relay :25 (oss-oo_mail)	blocked
postgres :5432 (oss-oo_default)	blocked
oss-oo.io:443 public (via NAT)	reachable (needed to clone/publish)
host gateway ports (e.g. git-ssh :2222)	reachable (public services)

Docker's inter-bridge isolation (DOCKER-ISOLATION iptables rules) walls job containers off from the internal compose networks — that is the load-bearing wall and it holds today. Egress to the internet is open (NAT): jobs can fetch dependencies, which also means they can mine, spam, or exfiltrate — see §6.

Residual risks that no bridge isolation covers:

• The runner daemon is root with docker.sock — the runner itself is trusted (pinned image, our registration); a runner compromise = host compromise. Job containers do NOT get the socket (docker_host: "-" default).
• Workflows can pick any container image (container: image: in the workflow overrides the label image). Any image from any registry runs with the same sandboxing as label images. Mitigation = caps + timeout + egress posture (§6), not image pinning.
• Host ports are reachable from job networks (2222/80/443) — these are public services anyway; keep it that way (never bind a private service to the host bridge gateway).

3. Hardening toolbox (forgejo-runner v13 + instance side)

Runner config (config.yml, generated reference at code.forgejo.org/forgejo/runner — internal/pkg/config/config.example.yaml; the historical public-box configuration remains in runner-box/config.yml):

Option	Default	Hardened setting for a public runner
runner.capacity	1	1 (one public slot; raise only with headroom)
runner.timeout	3h	inherit the pinned runner default, matching the PVE public lane
runner.shutdown_timeout	3h	10m — bounded graceful drain on stop
runner.labels	from .runner	oss-oo-public:docker://docker.io/library/node:24-bookworm (fully-qualified pin)
container.network	"" (per-job)	keep "" (ephemeral per-job networks + isolation §2) or a dedicated internal: true network (§6 egress options)
container.privileged	false	false
container.options	—	--memory=1g --cpus=0.75 --pids-limit=256 --security-opt no-new-privileges (job-container resource caps)
container.valid_volumes	[]	[] (no host mounts into public jobs)
container.docker_host	"-"	"-" (never expose the socket)
container.force_pull	false	true for the public runner (fresh image, no stale tag cache)
cache.enabled	true	false for the public runner (cache poisoning is a documented Forgejo attack vector + disk growth)

The daemon ceiling also applies when a workflow requests more time. On 2026-10-26 two healthy functional suites were cancelled by the older box's 30-minute override; the same head completed on the PVE public lane in just over 32 minutes. The older box was aligned to the same 3-hour default before its retirement. The PVE runner retains that pinned native default. This removes the inconsistent local override; it does not disable the native ceiling. In v13.2.0, setting timeout: 0 or a negative duration also retains that default. Workflow and server deadlines are separate and may be lower. Check all effective deadlines when diagnosing a cancellation, and review them when upgrading runners. A successful whole job is required; partial passing output is not a substitute. A quiet log by itself is not evidence that a job or runner has stopped working.

Instance side (app.ini / env, applies fleet-wide — our own repos too):

• [actions] LOG_RETENTION_DAYS / ARTIFACT_RETENTION_DAYS (defaults 365 / 90): every public job stores logs and artifacts on the forge data volume for that long. For the public era: 30 / 14 days is plenty for a free tier (tighten in the Step B PR; verify effective defaults at rollout).
• Per-repo and per-org Actions unit toggles exist today (repo settings; org default units) — the moderation lever, see §5.
• Upstream quota engine (disk soft-quota shipped; per-user CI build minutes in development) — nothing stable to build on yet; we are our own upstream and can adopt or patch later once it matures.

4. Who runs on what — label architecture and runner scoping

Forgejo schedules a job to any runner whose labels match runs-on:; a runner serves exactly the labels it registered. Labels alone are NOT an access boundary — verified live 2026-09-23 (§4.1): a throwaway user's repo requesting runs-on: oss-oo-internal was dispatched to a trusted internal runner within one second. The load-bearing gate is runner registration scope (native Forgejo): a runner registered with an org-scoped registration token (owner_id set on the registration) only ever receives jobs from that org's repositories — regardless of which labels a stranger requests. Current fleet:

• Internal lane, org-scoped: oss-oo-runner-1 (registration 11) is registered to OSS-OO; oss-singularity-internal-pve1-1 (registration 14) to OSS-Singularity. Both run in separate projects on PVE VM1005. Each org has its own slot — laboratory CI and website CI no longer queue behind each other. Label on both: oss-oo-internal:docker://docker.io/library/node:24-bookworm. Registration 14 also retains its unique admission label oss-singularity-internal-pve1; its organization scope is unchanged.
• Native-test lane, repository-scoped: registration 13 serves only its admitted repository with commons-native. Its separate execution policy and successful native tests do not change either general internal lane.
• Public lane, instance-scoped: oss-oo-runner-public-pve1-1 on PVE VM1003 serves only oss-oo-public; every account's repo may request it. The older oss-oo-runner-public-1 registration has been retired.
• The legacy docker label is retired fleet-wide; a repo writing it finds no runner (by design).
• GitHub-default labels (ubuntu-latest, ubuntu-24.04, …) never had a runner either: workflows requesting them wait forever and pile up zombie runs (22 in the zcode-dispatch repos, cancelled 2026-09-24). Forge-side workflows must request oss-oo-public / oss-oo-internal. The source at 2e0ea7cea712d468278d30f265c6a53d0305bba1 (modules/actions/workflows.go, ListWorkflows) selects the first existing directory in .forgejo/workflows, .gitea/workflows, .github/workflows order. It does not merge files by basename. Even an empty .forgejo/workflows directory suppresses later fallbacks; the adjacent upstream unit tests explicitly cover this behavior. Put every intended Laboratorium workflow in .forgejo/workflows/. Repositories keeping a GitHub mirror can retain its separate config, but a .github file without a same-named native file does not run here when the native directory exists. This corrects the previous per-basename claim; it does not establish the cause of historical runs. The source/version correspondence is not a deployed-image attestation.
• User-facing docs for contributors: the OSS-OO/ci-examples repository (label usage, caps, fair-use rules, copy-paste workflows, proven green on the public lane).

4.1 Label-collision finding & fix (2026-09-23, release-hardening sweep)

• Verified: push to a throwaway non-member repo with runs-on: oss-oo-internal → task 197 executed on oss-oo-runner-internal-2 (trusted, uncapped default config) — the label string is requestable by anyone and labels were the only split.
• Fix (native, no fork patch): both internal runners were re-registered with org-scoped tokens minted via GET /api/v1/orgs/{org}/actions/runners/registration-token (org owner). Dispatch-side scope filter confirmed in source (GetAvailableJobsForRunner: owner_id != 0 restricts candidate jobs to that owner's repositories).
• Proof both directions: post-fix, the same throwaway repo's internal job stayed waiting forever (no runner can take it) while its oss-oo-public control job ran green; real workloads proven too — website ci.yml re-run green on the OSS-Singularity-scoped runner, laboratory CI green on the OSS-OO-scoped runner.
• Watchdog: scripts/runner-box-watch.sh alerts when a foreign-repo job requests oss-oo-internal (it will wait forever — that is now an abuse/accident signal, not a hole) or when one of OUR internal jobs waits >2 h (outage signal).
• Rollback anchors: pre-swap .runner files are kept beside the live ones on each box (/data/.runner.bak-instance-scope in the runner volumes); restoring one + restarting the daemon reverts to instance scope (old registrations would need re-minting after 2026-09-23 row cleanup).

5. Who gets public CI — options for the owner

Native capability facts (verified, not invented): Actions can be toggled per repo and per org default-units today; global on/off via [actions] ENABLED; per-user CI-minute quotas do NOT exist upstream yet (§3). Options, from most open to most controlled:

1. All email-confirmed users, Actions ON by default. Zero admin overhead, maximum welcome. Abuse surface = anyone with a confirmed mailbox; contained by §6 caps + label split + abuse response. Recommended start for the "public - free!" promise, paired with option 3's moderation lever in the back pocket.
2. Per-repo opt-in (Actions OFF by default for new repos). New repos start without Actions; users request the flip (issue to owners / future automation). Strongest moderation without quotas; costs a manual step and friction against the open-door ethos.
3. Org members only / selected orgs. Public CI as a privilege of the orgs we curate (OSS-OO, OSS-Singularity). Simple, but excludes exactly the outside contributors public CI is meant to serve.
4. Hybrid (recommended): option 1 at launch with §6 hardening + monitoring; if abuse appears, degrade to option 2 by flipping org default units (one setting) and keep trusted repos enabled.

Rented capacity ("kaufbare" step, Step C) can layer a premium label (e.g. oss-oo-public-fast) later — same gating question returns then.

6. Abuse vectors & countermeasures

Vector	Countermeasure
Crypto mining (CPU)	--cpus=0.75 cap per job, 20m timeout, capacity 1 (one abuser max), queue visible in forge admin; 2 vCPU host keeps forge responsive
Memory exhaustion (OOM host)	--memory=1g per job; ~2.9 GiB available + 1 job max; --pids-limit=256 (fork bombs)
Disk fill (images, workspaces, logs, artifacts)	no cache (poisoning + growth), force_pull, log/artifact retention 30/14d, disk alarm at 70 % exists, docker system prune in RUNBOOK incident table; job workspaces live in the runner volume (observable, one docker rm away)
Egress spam / exfil from our IP	THE open risk of increment 1 (egress is NAT-open): caps + timeout + label split + abuse response (takedown, runner kill: it runs in its OWN compose project → docker stop without touching forge). Harder walls = §6.1 options, owner picks
Fork-PR untrusted code	Verified trust model: pull_request from forks runs the PR's workflow WITH NO repo secrets and a read-only automatic token (safe by design); pull_request_target runs base-branch workflows WITH secrets — our repos must never check out PR heads in pull_request_target workflows (policy note; grep our repos at Step B)
Secret exposure on runners	No instance secrets in public-lane envs; runners carry only their registration secret (.runner file, volume-internal); public jobs get at most the read-only fork token
Malicious job image (container: image:)	Inherent to the docker executor; contained by the same sandbox (§2) + caps + timeout; egress wall (if chosen) is the real answer; force_pull avoids stale-tag tricks
Runaway queue / job storms	capacity 1 per public runner; queue depth visible in forge admin; monitoring hook (§8) mails on sustained backlog

6.1 Egress posture options (owner decision for Step B)

1. Open egress + caps (recommended increment 1): jobs can fetch dependencies (npm, pip, apt) — the runner is actually useful. Risk: our IP's reputation (spam) and mining-pool egress; mitigated by CPU/time caps and instant kill switch. Free.
2. Blocked egress (internal: true docker network for job containers): jobs lose internet (only oss-oo.io via explicit exception is complex) — kills mining/spam utterly but also kills dep-fetching; public CI becomes near-useless for real projects. Free. Fallback if abuse hits before better tooling exists.
3. Allowlist egress proxy (follow-up, Step C era): dedicated runner box routes job egress through a filtering HTTP(S) proxy (package registries + forge only). The honest long-term answer; needs the dedicated box (isolation of failure domains) + config work. Costs the Step C box.

7. Capacity math & Step B recommendation (no new spend)

CX23 budget: 3.8 GiB total, ~2.9 GiB available idle, keep ≥1 GiB headroom (spikes, page cache, backups) → ~1.9 GiB CI budget.

• Internal lane job peaks ≈ 1.2–1.5 GiB (website Foundry+Node build); laboratory CI ≈ 0.1 GiB.
• Public lane job capped at 1 GiB (--memory=1g), worst case both lanes building simultaneously: ≈ 0.9 (stack) + 1.5 (internal) + 1.0 (public) = 3.4 GiB < 3.8 GiB — tight but safe, and both-full is rare; the 0.75 CPU cap leaves 1.25 vCPU for forge + internal lane.

Step B as built (2026-09-23, owner GO + hardware override): the co-located variant below is superseded — public jobs run on a dedicated box so they can never contend with the forge stack:

• Box oss-oo-runner-1: Hetzner cx33 (4 vCPU / 8 GB / 80 GB, fsn1, €8.49/mo net — under the approved cpx21 frame because cpx21 is deprecated/unorderable in the EU since 2025-12; verified live at order time), docker-ce app image, user runner, Hetzner firewall (inbound SSH + ICMP only). Station alias oss-oo-runner, forge→box alias oss-oo-runner-box (see RUNBOOK).
• Runner oss-oo-runner-public-1: compose project oss-oo-runner from this repo (runner-box/), full §3 hardening matrix (runner-box/config.yml), label oss-oo-public only, registered with a CLI-minted token that never left the server/box shells.
• Internal lane: existing runner re-registered to oss-oo-internal (dual-label first so no CI broke mid-migration, docker label dropped after all our workflows moved). The singularity foundation box's runner (which had been sharing load under the legacy docker label with DEFAULT config next to the publication engine) was re-registered as oss-oo-runner-internal-2 in the same window — the legacy label is extinct fleet-wide and the internal lane gained a second slot (forge + foundation box). Later the same day both internal runners were org-scoped (§4.1) — OSS-OO and OSS-Singularity each own exactly one internal slot; instance-scoped dead registrations removed.
• Instance side: [actions] LOG/ARTIFACT_RETENTION_DAYS = 30/14.
• Monitoring: scripts/runner-box-watch.sh on the forge (cron 15 min: box reachability, stuck jobs >1 h, disk ≥70 %) through the ops-mail channel; kill switches in the RUNBOOK fleet section.

Original (superseded) co-located design, kept for the record: CX23 budget math — 3.8 GiB total, ~2.9 GiB available idle, ≥1 GiB headroom → ~1.9 GiB CI budget; internal job peaks ≈ 1.2–1.5 GiB; a public job capped at 1 GiB fit alongside (worst case ≈ 3.4 GiB). The owner override removes this coupling entirely; the numbers remain valid as the argument for WHY public jobs got their own iron.

8. Monitoring & operations (fleet section of the RUNBOOK, lands with Step B)

• Queue/job pulse: small server-side cron (forge user, alert-mail channel) that counts running job containers and mails when the public slot is busy >N consecutive checks or a job approaches the timeout — reuses the existing ops-mail path, no new services.
• Kill switches: public runner is one docker stop (own project); whole fleet = [actions] ENABLED=false + runner stops (forge stays up). Both documented next to the existing "Runaway CI runner" row.
• Restart discipline (unchanged): runner restarts only with an idle Actions queue; the 502-limbo release procedure stays in the RUNBOOK.
• Disk: 70 % alarm already live; public-lane disk budget = runner volume + images + 14d artifacts — covered by the same alarm.

9. Step C — scale-out (separate round, owner GO, costs money)

Live fsn1 pricing via Hetzner API (net, 2026-09-23 — cpx21 is deprecated in the EU (unavailable since 2025-12-31, US locations only) and was replaced by cx33 at order time):

Type	vCPU / RAM / disk	€/mo net	€/h net	Fit
cx23	2 / 4 GB / 40 GB	5.49	0.0088	the forge box itself
cx33	4 / 8 GB / 80 GB	8.49	0.0136	the public runner box (as built, Step B)
cpx11	2 / 2 GB / 40 GB	5.49	0.0088	smallest burst box
cax21 (ARM)	4 / 8 GB / 80 GB	10.49	0.0168	best €/core, ARM image caveats
cpx31	4 / 8 GB / 160 GB	17.49*	0.0280	sustained load + egress-proxy (*stale quote — verify)

Two shapes, combinable:

• Dedicated always-on runner host (recommended when public CI is routinely busy or before raising capacity/egress posture): cpx21 in FSN1, docker + forgejo-runner only, same §3 config with higher capacity (2–3), registered to oss-oo.io over HTTPS. Nothing stateful worth backing up (jobs are ephemeral; registration secret is re-mintable) — a rebuild script replaces backup. Monitoring: same cron pattern + the status-page "CI runners" component.
• On-demand burst box (scripts/runner-burst.sh, future): create via api.hetzner.cloud with the station token → cloud-init installs docker + runner and registers via a freshly minted registration token → hourly billing caps cost (cpx21 ≈ €0.37/day at 24h; burst windows cost cents) → idle-shutdown (no job for N minutes → deregister + delete server). Cloud-init + firewall (SSH only from station/forge) documented before first use; registration tokens are secrets and stay server/box-side.

Trigger for Step C: public queue latency hurts (jobs waiting >10 min routinely), sustained CPU saturation, or the wish for premium/rented labels ("kaufbare" capacity) — each its own owner GO.