Split the clean cache composition by origin: upload remnants, read cache, unknown #19

Merged

ClaudiuSchuster merged 4 commits from oss-oo/cache-origin-rt20261009 into main

2026-10-09 20:17:10 +00:00

ClaudiuSchuster commented

2026-10-09 20:15:41 +00:00

Owner

What this changes

rclone's VFS cache marks a successful upload's local copy "clean" —
indistinguishable from a copy created by reading/downloading. The Control
Center's cache composition therefore labeled post-upload remnants as "read
cache", which is semantically false (they never served a read) and hid what
the cache actually holds. This round makes the composition tell the truth
about origins:

Origin journal (app-side, additive): a small journal in the state
dir records paths that transitioned dirty → clean via a completed
upload, built on the existing upload-completion parsing in the state
aggregation (no new live-system probes, no rclone/mount changes).
Atomic append, 0600 file under a 0700 dir, byte-capped rotation sized
to cover the ~8.5-day cache-max-age horizon, corrupt-line tolerant
reads, sanitized relative paths only. Clean entries not journaled as
upload remnants count as read cache (the complement).
Migration honesty: clean entries already in the cache at upgrade
time have unknown origin and get their own bucket ("origin unknown" /
„Herkunft unbekannt") instead of being mislabeled; the bucket drains
naturally as the age policy evicts.
State schema (additive, version bump): the vfs/cache section splits
clean into upload_remnants_bytes/files, read_cache_bytes/files and
unknown_origin_bytes/files; older snapshots without the fields default
the whole clean population to read cache (fixtures + UI consumers
updated together).
UI: the cache composition bar splits the neutral segment into two
neutral-family tones (upload remnant vs read cache) plus a muted
third variant for unknown-origin that visually drains; per-segment
tooltips EN + DE explain origin semantics plainly; the sub-line gains
per-origin counts/bytes at full width with graceful minimum-width
degradation; card-level delegation tooltips stay intact; zero segments
render nothing.
Docs: the OPERATIONS cache section and card description document
each origin's meaning, the journal basis and the unknown bucket's
lifecycle.

No eviction-policy changes (the 0.9.6 early tier targets verified copies
regardless of origin — untouched), no new settings.

Evidence

Independent validation: ACCEPT-WITH-FINDINGS — all six items PASS
(journal mechanics incl. rotation math vs append rate, migration honesty
via the unknown bucket + complement rule, additive schema with legacy
default, UI split/tooltips/degradation/zero-collapse, docs, non-goals);
make check green; git diff --check clean; laboratory remote branches
exactly ['main'] (branch-hygiene assert).
Gates: the validator's single gate finding (prettier column-padding
on one reworded table line, whitespace-only) is fixed at source in this
branch (fd2298f); changed Markdown now passes prettier@3.5.3 --check;
git diff --check clean.
Tests (+308 across state/UI): journal append/read/rotate/tolerance;
remnant/read/unknown split fixtures; legacy-snapshot default; EN/DE
tooltips; min-width degradation; zero-collapse per segment.

Non-blocking findings (recorded; from the validation dossier)

N2: missing suite leg for append-side torn-tail protection (the probe
covered the behavior; a permanence leg writing a journal without a
trailing newline before an append would pin it).
N3: theoretical concurrent-run rotation race — two simultaneous
pdrive-state invocations could interleave read→rotate→replace and lose
a just-appended entry (duplicates are harmless; impact bounded to
under-classifying a remnant as read cache until re-observation or aging
out; the journal is a classification aid by design).
Design observation (documented in code+docs): a mount-log-only completion
journals size 0 — the file counts as a remnant while its bytes land in
the read-cache complement until a sized record appears; per-bucket clamps
keep the sums exact either way.

## What this changes rclone's VFS cache marks a successful upload's local copy "clean" — indistinguishable from a copy created by reading/downloading. The Control Center's cache composition therefore labeled post-upload remnants as "read cache", which is semantically false (they never served a read) and hid what the cache actually holds. This round makes the composition tell the truth about origins: 1. **Origin journal (app-side, additive):** a small journal in the state dir records paths that transitioned dirty → clean via a completed upload, built on the existing upload-completion parsing in the state aggregation (no new live-system probes, no rclone/mount changes). Atomic append, 0600 file under a 0700 dir, byte-capped rotation sized to cover the ~8.5-day cache-max-age horizon, corrupt-line tolerant reads, sanitized relative paths only. Clean entries not journaled as upload remnants count as read cache (the complement). 2. **Migration honesty:** clean entries already in the cache at upgrade time have unknown origin and get their own bucket ("origin unknown" / „Herkunft unbekannt") instead of being mislabeled; the bucket drains naturally as the age policy evicts. 3. **State schema (additive, version bump):** the vfs/cache section splits clean into `upload_remnants_bytes/files`, `read_cache_bytes/files` and `unknown_origin_bytes/files`; older snapshots without the fields default the whole clean population to read cache (fixtures + UI consumers updated together). 4. **UI:** the cache composition bar splits the neutral segment into two neutral-family tones (upload remnant vs read cache) plus a muted third variant for unknown-origin that visually drains; per-segment tooltips EN + DE explain origin semantics plainly; the sub-line gains per-origin counts/bytes at full width with graceful minimum-width degradation; card-level delegation tooltips stay intact; zero segments render nothing. 5. **Docs:** the OPERATIONS cache section and card description document each origin's meaning, the journal basis and the unknown bucket's lifecycle. No eviction-policy changes (the 0.9.6 early tier targets verified copies regardless of origin — untouched), no new settings. ## Evidence - **Independent validation:** ACCEPT-WITH-FINDINGS — all six items PASS (journal mechanics incl. rotation math vs append rate, migration honesty via the unknown bucket + complement rule, additive schema with legacy default, UI split/tooltips/degradation/zero-collapse, docs, non-goals); `make check` green; `git diff --check` clean; laboratory remote branches exactly `['main']` (branch-hygiene assert). - **Gates:** the validator's single gate finding (prettier column-padding on one reworded table line, whitespace-only) is fixed at source in this branch (`fd2298f`); changed Markdown now passes `prettier@3.5.3 --check`; `git diff --check` clean. - **Tests (+308 across state/UI):** journal append/read/rotate/tolerance; remnant/read/unknown split fixtures; legacy-snapshot default; EN/DE tooltips; min-width degradation; zero-collapse per segment. ## Non-blocking findings (recorded; from the validation dossier) - N2: missing suite leg for append-side torn-tail protection (the probe covered the behavior; a permanence leg writing a journal without a trailing newline before an append would pin it). - N3: theoretical concurrent-run rotation race — two simultaneous `pdrive-state` invocations could interleave read→rotate→replace and lose a just-appended entry (duplicates are harmless; impact bounded to under-classifying a remnant as read cache until re-observation or aging out; the journal is a classification aid by design). - Design observation (documented in code+docs): a mount-log-only completion journals size 0 — the file counts as a remnant while its bytes land in the read-cache complement until a sized record appears; per-bucket clamps keep the sums exact either way.

ClaudiuSchuster added 4 commits

2026-10-09 20:15:41 +00:00

Journal upload completions and split clean cache by origin 2019498c53

pdrive-state now keeps an additive upload-origin journal
(pdrive-cache-origin.log) in the state directory, following the
graph-history log pattern: mode-0600 file under the mode-0700 state
directory, single O_APPEND writes with torn-tail protection, atomic
byte-capped rotation that always retains the baseline line, and
corrupt-line tolerant reads. Upload completions reuse the transfer
history's existing rclone and mount-log parsing; no new live-system
probes are added.

The first run with live VFS statistics anchors the current clean
population as the migration baseline. Clean copies already cached at
that moment surface as their own unknown-origin bucket that drains
linearly over the configured retention instead of being lied into
either class. Newest-per-path journal entries younger than their
eviction horizon (the retention, or the shorter large-copy tier for
entries of known 1 GiB-or-larger size) count as upload remnants; every
remaining clean byte is read cache by complement, so downloads need no
journal.

The vfs section gains upload_remnants/read_cache/unknown_origin
bytes+files additively (schema version 3); each bucket is clamped so
the three origins always sum exactly to the clean population.

Split the cache composition bar by clean-copy origin 2242d84f43

The VFS-Cache card no longer labels every clean byte "read cache". The
composition bar's neutral segment splits into two neutral-family tones
— light blue for upload remnants that never served a read, deeper blue
for read-cache copies — while the draining migration bucket renders as
a muted gray unknown-origin segment. Each segment carries its own
tooltip in English and German, the card-level tooltip delegation stays
intact, and a caption sub-line below the bar repeats per-origin counts
and bytes, ellipsizing gracefully at the minimum window width.

Snapshots from an older pdrive-state without the origin fields count
the whole clean population as read cache, so legacy behavior is
preserved. The card detail line, the What's-happening cache line, the
transfers cache section heading and the clean row now use origin-neutral
"clean copies" wording, with the clean row's hover text stating the
live origin split. The demo state shows one remnant beside one
read-cache copy.

Document clean-copy origins in the cache documentation 3768489080

OPERATIONS explains the three clean origins (read cache, upload
remnants, unknown), the journal basis — one mode-0600 state-dir file,
appended from the already-observed completion records, rotated at a
byte cap covering the retention horizon — and the lifecycle of the
unknown bucket after an upgrade, including the honest limits of the
classification and that no eviction behavior changes. The VFS-Cache
card description matches the new bar, caption and legacy-snapshot
default, and EVERYDAY_USE names the origin split. The AGENTS.md source
map notes pdrive-state's single additive write.

Align the retention table to prettier style

Checks / Functional checks (pull_request) Successful in 1m23s

Details Explain this check

fd2298f1ac

ClaudiuSchuster merged commit 601484ae21 into main

2026-10-09 20:17:10 +00:00

ClaudiuSchuster deleted branch oss-oo/cache-origin-rt20261009

2026-10-09 20:17:10 +00:00

ClaudiuSchuster referenced this pull request from a commit

2026-10-09 20:17:12 +00:00

Split the clean cache composition by origin: upload remnants, read cache, unknown (#19)

ClaudiuSchuster referenced this pull request