OSS-OO/laboratory
OSS-OO/laboratory
Watch
5
0
Fork
You've already forked laboratory
 0
Code Issues 7 Pull requests 5 Projects Releases Packages Wiki Activity Actions
laboratory/docs/IGNITION.md
Claudiu Schuster a35da3623f
All checks were successful
ci / ci (push) Successful in 23s
Details Explain this check
Route the laboratory through PVE ingress (#125) 
Use the private PVE backend behind ingress-owned TLS, require exact deployment image selectors, and document the accepted operating and recovery paths. Full native VM1005 E2E, independent source review, public protocol acceptance and exact-head CI run277 passed.
2026-10-13 18:22:47 +00:00

5.3 KiB
Raw Permalink Blame History

IGNITION — from zero to https://oss-oo.io

Historical direct-edge VPS bootstrap, retained as source history. This is not the current PVE installation, update or rollback procedure. Do not run its bootstrap account creation, .env replacement, incomplete Compose startup, old DNS rollback or server-destruction steps on the restored system. Use RUNBOOK: Current deployment and PVE HTTP backend instead. Current production writes require explicit reconciliation before reverting to old application data; the historical redirect host is not a data rollback.

For the current cross-product architecture, verified infrastructure and migration gates, see the Arkenstein handbook.

Two humans-actions are required (server + nothing else; DNS is already automated via the Cloudflare token in the station keyring). Everything else is scripted. Total time on a fresh server: ~10 minutes.

1. Buy the server (operator)

Hetzner Cloud → new server:

Setting Value
Type cx23 (x86, 2 vCPU / 4 GB / 40 GB, 5.49 €/mo)
Image Ubuntu 24.04
Location Falkenstein (FSN1) or Nuremberg (NBG1)
SSH key ~/.ssh/oss_oo_forge_ed25519 on the station

API note (2026): the Cloud API base is api.hetzner.cloud — the old api.hetzner.com paths 404. Full token: ~/.api-tokens/hetzner_api.token. The agent can create the server itself with that token (done for oss-oo-forge-1 on 2026-09-22).

Alternative ARM: cax21 (4 vCPU / 8 GB / 80 GB, ~10.49 €/mo) — all pinned images are multi-arch; x86 keeps third-party CI images friction-free.

Orgs on the forge are created by the OWNER's personal account (not the bootstrap admin); scripts/bootstrap-forge.sh creates only admin + CI runner unless org names are passed explicitly.

Hand the IP (and root SSH, if not key-injected) to the agent. Fully automated alternative: create a Hetzner Cloud API token (read/write) — then the agent provisions the server itself.

2. Provision (agent, ~2 min)

ssh root@SERVER_IP 'PROVISION_SSH_KEY="$(cat ~/.ssh/oss_oo_forge_ed25519.pub)" bash -s' \
  < provision/provision.sh

Installs docker + compose, ufw (22/80/443/2222), unattended-upgrades, forge ops user, key-only sshd, /opt/oss-oo.

3. Ship the stack (agent)

rsync -a --delete --exclude .git --exclude .env --exclude backups/ \
  /home/claudiu/git/oss-oo/ forge@SERVER_IP:/opt/oss-oo/

ssh forge@SERVER_IP
cd /opt/oss-oo
cp .env.example .env
# set POSTGRES_PASSWORD, FORGEJO_ADMIN_PASSWORD (openssl rand -hex 24),
# and ACME_EMAIL (runner image is pinned already)
docker compose -f compose.yml -f compose.prod.yml up -d
exit
./scripts/bootstrap-forge.sh          # admin, orgs, CI runner

4. DNS flip (agent, automated)

oss-oo.io lives on Cloudflare DNS; the station keyring token has DNS edit on the zone. Currently the A record points at 188.68.47.93 (netcup redirect to oss-singularity.io) — that becomes the rollback target.

  • Set A oss-oo.io → SERVER_IP (DNS-only/grey cloud, no proxy: git-over-ssh on 2222 and webhooks do not tolerate CDN proxying)
  • Set A www.oss-oo.io → SERVER_IP (Caddy 301s www → apex)
  • Wait for propagation, then Caddy obtains the Let's Encrypt certificate on first HTTPS hit.

5. Verify (agent)

./scripts/verify-live.sh oss-singularity/website SERVER_IP

Checks DNS, HTTPS edge, forge API, www redirect, git ls-remote, TLS cert.

6. Operate

  • Backups: install the cron line from scripts/backup.sh header (root).
  • First repos: push mirrors from /home/claudiu/git/mirrors/ (see scripts/mirror-github.sh) into the forge orgs, then retarget checkouts. Done 2026-09-22: all 16 oss-singularity repos live in the OSS-Singularity org (heads + tags; forgejo rejects GitHub refs/pull/* as hidden refs, so push-mirrors.sh pushes explicit refspecs).
  • This repository is self-hosted at https://oss-oo.io/OSS-OO/laboratory — main is protected there (PR + green CI required), making oss-oo.io self-hosting from day one.
  • Independent watchdog: follow the observer contract. Use shadow until the separate notification evidence, stopped/finished/reconciled prior owner and fresh eight-key baseline are verified. Source tests or an ingest ACK alone do not authorize cutover or prove DOWN/recovery email delivery. Before an observer source upgrade, verify the restricted systemd credential ACL/mount context with a disposable credential and the exact accepted reader; preserve that no-network proof before a silent shadow run. Credential compatibility alone does not authorize a timer, notification or owner change. Direct HTTP workers in a declared service context must also refuse an inadmissible required MQTT credential before any outgoing observation.

Rollback

  1. DNS: restore A oss-oo.io → 188.68.47.93 (netcup redirect). Propagation minutes-to-hours depending on TTL.
  2. Server: docker compose -f compose.yml -f compose.prod.yml down — data stays in volumes; server can be destroyed any time after backups exist.