laboratory/docs/INFRASTRUCTURE_ROLLOUT.md

Infrastructure rollout — planned VM roles and acceptance gates

This document turns the approved Arkenstein migration direction into a staged, checkable rollout for the Laboratorium (oss-oo.io) and its companion product OSS Singularity (oss-singularity.io). The two products belong together but stay separately operable: each keeps its own publication gates and data, and they integrate only through versioned, deliberately coordinated interfaces.

Status reconciliation: 2026-10-17. The Laboratory cutover on PVE1 was accepted on 2026-10-13, 17:22 UTC. The dated current-deployment record in RUNBOOK binds its restore, transport, backup and recovery limits. The October 3 plan, budgets, ordered sequence and unchecked gate worksheet below are retained as Historical; they are not the current guest inventory and do not reopen accepted cutover. A checkbox is not retrospectively ticked without its own dated receipt.

Subsequent accepted work includes the separate website origin on VM1003, native Core d91fd57598366723548c6a4ca820182b5a79fb4e (2026-10-15), runner-cache work and infrastructure documentation milestone accepted on 2026-10-16. Website Run 260930000000012 completed on 2026-10-16 with rollback retained and admission closed; its roadmap entry links immutable public intent/outcome evidence. These are separate accepted deliveries, not a blanket completion of every role, recovery drill or later retirement gate. Host HA/PVE2+3, OVH and federated preservation remain later work with their own decisions; one hypervisor is still one failure domain.

Status labels follow the Arkenstein handbook: Verified, Historical, Prepared, Planned. The historical October 3 snapshot below retains its original labels within that dated scope. Companion documents: ARCHITECTURE_MAP (visual overview, five maps), IGNITION (historical initial installation), RUNBOOK — Backups (backup/restore implementation), RUNNERS (build execution).

1. Historical starting point (evidence date 2026-10-03)

Present and plan are kept strictly apart:

• Verified: the dedicated Proxmox VE host with a mirrored ZFS pool is built and privately administrable (WireGuard management, firewall, reboot-persistent). It runs no application guests.
• Verified (existence only): one isolated restore-drill guest (VM 9901) has been created for the restore rehearsal. The rehearsal itself is in progress; its intermediate installer state is not a result and is not published. A working installer is not a proof of restore, and the restore9901 installer is not a production template.
• Prepared: the independent observer host exists outside the main host's failure boundary; production monitoring has not moved to it.
• Planned: everything else in this document. No further guest exists, and no VM ID, address, network, bridge, VLAN or image beyond the rehearsal guest's setup is decided by this document. Guest IDs, networks and addresses are chosen from a fresh inventory at creation time, not reserved here.

The current ZFS mirror protects against a single disk failure on one physical host. It is not host high availability, not site-loss protection and not application-corruption protection. Off-site backups exist and a complete new backup set is verified — but a complete production restore is not yet proved. Backup success and restore capability are different gates.

2. Planned roles

Five persistent guest roles are planned. They are starting limits, not performance guarantees and not allocations already applied. The release-controller migration is deliberately last.

Role	Initial maximum CPU	Fixed RAM budget	Initial virtual disk budget	Responsibility
Public ingress	2 vCPU	4 GiB	32 GiB root	TLS and explicit routing for both products; no databases, CI or management plane exposure. Software selection follows the retained deployment/config inventory.
Laboratorium	6 vCPU	24 GiB	64 GiB root + 512 GiB data	Repository hosting and its product-owned database, package/LFS/attachment data and service dependencies. App and database stay co-located initially for a bounded, testable recovery unit; a later split follows measured demand or a second failure domain.
Singularity	4 vCPU	12 GiB	48 GiB root + 128 GiB data	Website origin and future product services. The website's current hosted edge worker/database (Cloudflare Worker/D1) remain their actual authority; a VM's existence does not migrate them and does not enable QA/Connect.
CI execution	8 vCPU total	24 GiB total	64 GiB root + 192 GiB disposable workspace/cache ceiling	Builds and tests, separated from product workloads, admin keys, publisher credentials and backup secrets. Totals apply across any temporary job guests.
Trusted release control	2 vCPU	8 GiB	32 GiB root + 64 GiB state	Protected publisher and durable acceptance/history. Never co-located with the ordinary CI executor.

Trust rules that are part of the decision:

• CI receives no publisher credentials, no backup secrets and no admin keys — a passing build never earns production authority.
• Untrusted jobs need a disposable guest boundary with an explicit egress and credential design. A container limit on a shared kernel does not satisfy this.
• The release controller is never co-located with the ordinary CI executor.

3. Capacity and reserves

The host is verified as 24 threads / 128 GiB RAM. The initial role totals (22 vCPU, 72 GiB RAM) stay inside it: 16 GiB are retained for the ZFS ARC and 8 GiB for the host and its services, leaving 32 GiB RAM headroom — within which the 12 GiB restore guest fits. Persistent virtual disk budgets total 1136 GiB; the restore rehearsal's subtree (~160 GiB) is separate and temporary.

• vCPU allocations cap scheduling intent; they do not reserve physical cores. CPU and IO contention need measured caps/weights.
• Fixed memory and bounded disk/IO budgets are what stop a runaway CI job from consuming the whole host by accident.
• Sparse volume capacity is not free physical capacity. Quotas, actually allocated bytes, snapshots and peak restore/copy space all count. Growth decisions check the live pool and the off-site retention/restore capacity first.
• Backup archives, restore staging and build scratch never land on the hypervisor root or on product system disks. Small bounded operator journals and rollback configuration are the only operator files kept there, and they are not backups.

4. Ordered rollout

Historical sequence from 2026-10-03, retained for its acceptance contract. The stages are strictly ordered; each begins only when the previous stage's evidence exists. The restore9901 installer is not a production template — production guests come from an independently accepted production bootstrap.

1. Restore rehearsal (in progress). Finish the real networkless restore base/tools/password-login proof, then bounded transport/data download, then complete offline preservation checks for database, files, images and application state. A successful installer infers no production migration.
2. Production bootstrap. Provision the ingress and Laboratorium guests from an independently accepted production bootstrap. Prove complete restore, compatible configuration, private smoke tests, resource and backup limits, and rollback.
3. Cutover, one product at a time. Take a fresh off-site backup with the required consistency boundary immediately before cutover. Execute one named product cutover with health/API/Git/LFS/packages/ attachments/broker checks, a clear rollback deadline and the retained old source. No simultaneous website release or provider change.
4. Singularity. Stand the new website origin up on its own guest and test it privately — including the protected release/rollback path — before any public cutover; public bytes/headers are verified only after the actual switchover. No simultaneous product/provider change and no feature enablement ahead of its gate. Any future Worker/D1 move is a separate data/API project. Source publication, QA and identity activation gates stay intact.
5. CI. Replace the existing runner only after isolated jobs, exact native CI, resource ceilings, secret boundaries, cleanup and failure/restart tests pass. Never revoke the old runner before the replacement is observed end to end.
6. Release control (last). Transfer retained historical failure evidence, nonce/counter/key/locks, all policies/images/source and the append-only history exactly. No migration while a deployment is pending or ambiguous.
7. Retirement. Retire old resources only after the agreed observation/rollback period and a second independent recovery proof. Public capability claims are published only from accepted deployed evidence, separately from this plan.

Product contributions after accepted migration — 2026-10-17

The owner scheduled Laboratory #119/#120/#122, website #16 and agent-plugin #1 after migration on 2026-10-11. The migration is delivered. The post-migration intake now separates accepted SEO source/live work, the completed four reference redirects and Explore social fix, Google follow-up, and the plugin's open original-object transfer from its delivered public reader/client proofs. Do not repeat the accepted corrections or treat these as migration gates.

Guide, project brief and continuation Composer are shipped. The next actual contribution and resumable continuation proof, and native read-only Actions explanation from #122, are in progress rather than accepted. Optional native reruns still require their own server-side identity, permission, atomicity and recovery contract. Source review, CI, publication, client activation and external search processing remain separately evidenced outcomes.

5. Acceptance checklist

This is the retained October 3 worksheet, not a current list of undone migration work. The accepted cutover is recorded above; individual boxes remain untouched until reconciled to their own date and retained receipt.

Stage 1 — restore rehearsal

• Restore base runs without network and accepts password login; base tools present and identified.
• Bounded transport and data download complete with integrity checks.
• Offline restore covers database, files, images and application state; each component is compared against its backup manifest.
• The restored application starts without production credentials, outbound mail or public exposure, and what was and was not exercised is recorded.

Stage 2 — production bootstrap

• Ingress and Laboratorium guests exist, built from an accepted production bootstrap (distinct from the restore9901 installer).
• A complete restore of the verified production backup set succeeds in the Laboratorium guest.
• Configuration is compatible; private smoke tests pass for health, API, Git, LFS, packages, attachments and the broker.
• Memory, CPU, disk and IO limits are active and observed effective.
• Guest backup integration has a compatible destination and a preservation plan (not described as an installed backup server).
• Rollback from the guest state is rehearsed.

Network segmentation gates (before any production traffic on the new guests)

• Fresh interface, bridge, route, IPv4/IPv6, VPN and address inventory, checked for overlap with the workstation, the provider and the temporary restore networks; guest IDs and addresses are allocated only after this inventory.
• A concrete before/after configuration is reviewed, with a saved bounded rollback configuration and a recovery channel that survives a mistaken network rule; the existing private admin access survives the change.
• One bounded non-production network change is applied first, then allowed traffic and denied paths are proven from representative guest roles — including CI-to-management, CI-to-release, CI-to-database and cross-product database isolation.
• Source-address restrictions verified; IPv6 parity or deliberate disabling recorded; DNS and NTP behavior, package-update paths and certificate issuance/renewal verified.
• The routing/NAT owner and firewall backend are identified, and behavior during host boot/reload and on failure is verified — a working request to one service is not the isolation proof.
• The accepted network state is bound to each product's bootstrap/backup/restore/cutover plan; old services and rollback capacity are retained until their own acceptance gates close.

Stage 3 — cutover

• A fresh consistent off-site backup is taken immediately before cutover and its completeness is verified.
• Writers are fenced; the final consistent capture transfers and verifies.
• The named product cutover passes its health/API/Git/LFS/packages/ attachments/broker checks.
• The rollback deadline is defined and the old source is retained unchanged.
• No website release or provider change runs in the same window.

Stage 4 — Singularity

• The new origin serves from its own guest and passes private testing, including the protected release/rollback path, before any public cutover.
• Public bytes/headers verify only after the actual switchover to the new origin.
• No product/provider change runs in the same window and no feature is enabled ahead of its gate.
• Worker/D1 authority is unchanged; QA/Connect remain closed.

Stage 5 — CI

• Jobs run isolated on CI guests within their fixed budgets.
• Exact native CI parity with the Laboratorium's own CI lanes is proven.
• The secret boundary is audited: no publisher credentials, backup secrets or admin keys reachable from CI.
• Untrusted jobs run behind the disposable guest boundary with an explicit egress design.
• Cleanup, failure and restart tests pass.
• The old runner remains registered and serving until the replacement is observed end to end.

Stage 6 — release control

• State transfer is byte-bound and complete: historical failure evidence, nonce/counter/key/locks, policies, images, source and append-only history.
• The old controller's authority is closed.
• Release rehearsal and restore rehearsal pass on the new controller.
• No deployment was pending or ambiguous during migration.

Stage 7 — retirement

• The agreed observation/rollback period elapsed without unresolved findings.
• A second independent recovery proof completed.
• Old resources are retired and public capability claims updated from accepted evidence only.

6. Per-role boundaries

Role	Prerequisites	Responsible components	Evidence still required	Cutover / rollback boundary
Public ingress	Accepted production bootstrap; fresh guest ID/network/address from inventory; routing map reviewed against the retained deployment/config inventory	TLS and explicit routing for both products; no databases, CI or management plane	Route/firewall/TLS proofs and external health checks recorded; backup inclusion decided	Cutover bound to its product's stage-3 window; previous ingress retained until the rollback window closes
Laboratorium	Stage 1 complete; accepted production bootstrap	Repository hosting, product-owned database, package/LFS/attachment data, service dependencies	Complete restore of the verified production set; compatible configuration; private smoke tests; effective limits; rehearsed rollback	Fresh consistent off-site backup immediately before cutover; single named cutover; explicit rollback deadline; old host retained
Singularity	Protected release/rollback path healthy for the website	Website origin, future product services	Origin serving from the guest; protected release/rollback proven privately before any public cutover; public byte/header verification after switchover	Its own cutover window, never simultaneous with a release or provider change; Worker/D1 untouched
CI execution	Isolation, egress and credential design reviewed	Builds/tests on disposable, budgeted guests	Isolated jobs, native CI parity, ceilings under load, secret audit, cleanup, failure/restart tests	Old runner survives until the replacement is observed end to end
Trusted release control	All other roles accepted; byte-bound transfer plan	Protected publisher, durable acceptance/history	Exact state transfer; closed authority; release and restore rehearsal	Migration last; skipped while any deployment is pending or ambiguous

7. Planned network segmentation

The five roles need distinct communication policy. This is a root architecture decision (2026-10-03) and Planned throughout: no bridge, VLAN, guest address, forwarding rule or firewall configuration described here exists yet, and actual bridge IDs, subnets and VLANs are chosen only after a fresh inventory — they are not reserved or invented here. The existing host network and the working WireGuard administration stay unchanged during the restore installation.

Trust domains

• Management. Proxmox and privileged guest administration are reachable only through the reviewed WireGuard path and explicitly authorized administrator identities. Public management ports stay closed. CI, public ingress and ordinary product processes cannot reach hypervisor management.
• Public ingress. Only explicitly inventoried public service listeners, with reverse-proxy traffic to named product backends. No database, backup-repository or administrative API listener is exposed by this role. Each existing Git SSH, registry, API and event-bus entry point keeps its own protocol/port contract — moving HTTPS alone is not a complete migration.
• Product backends. Laboratorium and Singularity are separate trust domains. Their public APIs can integrate them; they do not gain direct access to each other's databases or data volumes. Initially each product's database stays local to its own recovery unit, without a broadly reachable database listener.
• CI execution. Disposable untrusted-job guests have no route or authority to management, privileged release control or backup repositories. Builds get only the repository/artifact endpoints, public dependencies and DNS/time service they require, under a reviewed egress policy. Untrusted pull requests receive no production, publisher, backup or admin credentials. A trusted runner supervisor's narrower role is distinct from code running inside a disposable job.
• Trusted release control. A separate execution and credential boundary from CI. It reads exact accepted source/check/artifact evidence and reaches only its scoped publication endpoints. Passing CI, a network connection or an artifact never grants release authority by itself; existing controller policies, counters, keys, locks and reconciliation gates still apply.
• Restore rehearsal (isolated exception). The restore guest stays NIC-less for the current base/tools proof. Later temporary download connectivity is a separate bounded lease to the exact off-site storage endpoint, with no access to products, CI or management; the NIC is removed again and the actual QEMU topology verified before restored database/application processes start. It never becomes a permanently connected production VM.
• External observer and off-site storage stay outside the main physical host. Application-specific backup writers retain only their required encrypted-backup rights; CI and public ingress receive none. Restore-read credentials and backup retention/deletion authority remain separate operational questions; read-only claims require actual server-side enforcement and a documented writer/prune fence.

Directional policy (default deny between trust domains)

Initiator	Destination	Allowed purpose, after exact endpoint review	Excluded authority
Approved administrator via VPN	Named management endpoints	PVE/private SSH and explicitly scoped maintenance	No public management listener or blanket CI-origin access
Public client	Named public ingress/service endpoint	The preserved HTTP(S), Git, registry, API or broker protocol as actually required	Database, host administration, backup repository
Ingress	Named Laboratorium/Singularity backend listener	Reverse proxy or explicitly inventoried service forwarding	Database/host/release administration; arbitrary private-network forwarding
Laboratorium	Singularity public API, when required	Versioned product integration with a scoped application identity	Direct Singularity database/filesystem access
Singularity	Laboratorium public API, when required	Versioned repository/identity/product integration	Direct Laboratorium database/filesystem/admin access
Disposable CI job	Approved repository/artifact/dependency endpoints	Checkout, dependency retrieval, scoped result production	PVE, product databases, publisher/backup credentials, arbitrary private destinations
Trusted release controller	Accepted evidence and scoped publication endpoints	The existing gated publication workflow only	General-purpose host administration or cross-product authority
Authorized backup writer	Off-site backup service	Direct encrypted backup stream under the retained backup policy	Local full-archive spooling or CI reuse of backup authority
External observer	Explicit public service/heartbeat/notification endpoints	Health observations and the reviewed independent alert route	Product/management mutation
Restore guest during a transport lease	One approved backup transport endpoint	Bounded verified download only	All other external/private paths; application execution while networked

Response traffic belongs to established permitted flows; it is not a general reverse-connection grant. DNS, NTP, package updates, certificate issuance/renewal and IPv6 each require explicit treatment. A broad private-address allow rule is not a substitute for this matrix, and the lack of a route alone is not a durable firewall policy.

flowchart LR
    admin["Administrator, VPN"] -->|"scoped PVE and private SSH maintenance"| mgmt["Management plane"]
    client["Public client"] -->|"HTTP(S), Git, registry, API, broker"| ingress["Public ingress"]
    ingress -->|"reverse proxy, named backends"| lab["Laboratorium backend"]
    ingress -->|"reverse proxy, named backend"| sing["Singularity backend"]
    lab <-->|"versioned public API, scoped identity"| sing
    ci["Disposable CI job"] -->|"repositories, artifacts, dependencies, DNS/NTP only"| deps["Repository and dependency endpoints"]
    release["Trusted release control"] -->|"gated publication only"| pubend["Scoped publication endpoints"]
    backup["Authorized backup writer"] -->|"encrypted stream"| offsite["Off-site storage"]
    observer["External observer"] -->|"health checks, alert route"| statusend["Status endpoints"]
    restore["Restore guest, transport lease"] -->|"bounded download only"| offsite

Implementation still to validate

Separate private L2 segments for the different trust domains, with explicit filtering at routing and guest boundaries, are the planned shape. A bridge name, a VLAN label or two guests sharing one bridge does not by itself prove isolation. The final implementation must identify the exact routing/NAT owner and firewall backend, the behavior during host boot/reload, and protection against unexpected guest source addresses or extra NICs; no host or guest gains implicit forwarding authority from an additional interface. The existing public management bridge is not the default network for product or CI guests, and no added physical switch uplink is implied by a private virtual bridge. The planned ingress can serve both products without granting them a shared database or management network. Private naming follows the DNS roadmap in cross-cutting boundaries; the working VPN bootstrap and the client default route/ordinary Internet DNS stay untouched by unrelated provisioning.

The open network gates are part of the historical acceptance worksheet.

8. Cross-cutting boundaries

Monitoring and alarms. Monitoring stays outside the main server. The accepted watchdog source is installed on the separate observer and has completed one silent shadow run; the cloud shadow timer stays disabled, the workstation timer stays active, and until the ownership handover is accepted the workstation observer remains the only active alarm sender. This planning work itself triggers no new alarm probe; later, separately ordered end-to-end acceptance runs remain their own gated changes, and none of the existing cutover gates is closed by prose. The observer cutover follows its own gates (silent shadow, preserved alarm state, exact routing, actual DOWN/recovery delivery, rollback) — a passed mail delivery test alone is not a passed automatic chain. A mail path through the monitored application host cannot be the only route for reporting its outage.

Private DNS (planned, later). The current workstation hosts-file entry that bridges the private management name is a working stopgap and stays until a replacement is proven. The later bounded block: define internal infra.oss-oo.io naming, address ownership and lifecycle; run an authoritative private zone with a VPN-reachable resolver and deliberate redundancy across failure domains (software is chosen after the topology is known, not assumed here); split DNS for the internal zone over WireGuard while clients keep their normal Internet DNS; avoid a cycle where VPN availability depends on its own unreachable resolver; keep service discovery, DNS availability and TLS trust as separate lifecycles; prove correct and negative lookups, reconnect/reboot behavior, resolver failure behavior and absence of DNS leakage; preserve a direct private-address/known-host recovery path. Hosts entries migrate only after working resolver/client tests and a documented rollback. No public admin port is opened for this.

Storage growth (planned, later). The verified ZFS mirror stays on the single physical host. Ceph is a future multi-host option, not a current one: it requires at least three suitable physical hosts, deliberate failure-domain placement, dedicated OSD disks, independent OS storage, CPU/RAM reserve and enough free capacity to recover after a host loss; Proxmox recommends at least 10 Gbit/s dedicated Ceph traffic (more for high-performance NVMe), and three replicas mean roughly one-third raw capacity before reserves. Three virtual nodes on one physical host are not three hardware failure domains. Hardware, network and pricing need fresh validation before any purchase. The current mirrored NVMe devices also host the installed PVE system and are never offered to a Ceph OSD wizard. Application state stays in explicit guest volumes so a later storage redesign can be planned and rollback-tested; introducing Ceph later would not require destroying ZFS, and Ceph does not replace backups. Kubernetes, additional guests and additional purchases are options under consideration, not decisions.

Publication gates. Existing website publication gates remain in force. The accepted website origin and later Run12 do not imply a Worker/D1 move, new identity authority or acceptance of every QA/Connect service. Guide, project brief and local Composer delivery are recorded above. Publisher/CI secret separation remains mandatory; any release-controller replacement needs its own exact transfer and recovery acceptance.