Share results anonymously: collection, outbox, sender, consent, ogc db measure #48

Manually merged

SkyfaR merged 7 commits from contrib into main 2026-10-07 10:58:26 +02:00

Conversation 0 Commits 7 Files changed 136 +10450 -286

SkyfaR commented 2026-10-07 07:43:59 +02:00

Owner

Copy link

Phase 3 of the community database design, on top of #47 (this branch contains it; merge #47 first).

Analysis prerequisites. The directory report splits bytes into compressible, No_COW and incompressible and aggregates per extension. With sharing on, a separate methods pass runs the samples already read through zstd 1/3/6/9/15 (and XPRESS 4K/8K/16K on Windows; LZX is measured only) at background priority, only when nothing else is asked; the visible estimate never waits for it. Levels 9 and 15 come from a deterministic subset of the samples (folders with ≥ 512 samples; every 8th sample of a file read whole, every 4th of a sampled file), scaled by the folder's ratio to level 6: CPU −68 to −80 % on real Steam folders, level 15 within 0.1–2.2 ‰ of the full estimate (owner's decision). The pass runs on AC power only (the timer's battery check; desktops count as AC); ogc analyze --on-battery overrides it. estimate::GENERATION is 2. The analysis cache keeps the estimator generation and size bucket. On Windows a measurement walk after a WOF run sums GetCompressedFileSizeW and checks every file's algorithm.

Contributions. collect.rs applies §3.6 (store key and build, fully installed, complete run, no exclusions or skip list, Windows measured only after a clean walk, not again within 180 days for methods already sent; excluded games are never shared). The outbox holds core values only, one item per game; the sender adds salt, PoW and padding, POSTs exactly 1024 bytes over a fresh ureq agent (no keep-alive, no TLS resumption, no cookies, constant User-Agent, SOCKS resolving remotely), random not-before 1–24 h, ≥ 2 h apart, ≤ 8 a day, only while the app is open or on ogc db send; the timer only queues. 202 accepted; every 4xx deletes; 410 turns sharing off; 429/503 back off; 14-day expiry; a 30-day pause after failures on at least 3 days.

UI. "Share results anonymously" switch with the §9.3 confirmation dialog, the one-time card after the first analysis, the outbox page (exact JSON, remove one, remove all), notices. CLI: ogc db outbox [--show] [--clear], ogc db send, ogc db measure (curated measurement in ogc-db's format, no paths or names; on Windows real WOF runs with --restore).

Review: 13 findings from two independent reviews (10 distinct, 1 high: Windows estimates and measurements replaced each other in the outbox), all fixed. Locally: fmt, clippy (both feature sets), 1850 tests incl. GUI under Broadway pass. Not run: the Windows GTK clippy (check-gui.sh) — CI covers it.

Fixed after the first CI run: the thread id comes from the system call instead of gettid(), which glibc 2.28 (the bundle's floor) lacks.

Phase 3 of the community database design, on top of #47 (this branch contains it; merge #47 first). **Analysis prerequisites.** The directory report splits bytes into compressible, No_COW and incompressible and aggregates per extension. With sharing on, a separate methods pass runs the samples already read through zstd 1/3/6/9/15 (and XPRESS 4K/8K/16K on Windows; LZX is measured only) at background priority, only when nothing else is asked; the visible estimate never waits for it. Levels 9 and 15 come from a deterministic subset of the samples (folders with ≥ 512 samples; every 8th sample of a file read whole, every 4th of a sampled file), scaled by the folder's ratio to level 6: CPU −68 to −80 % on real Steam folders, level 15 within 0.1–2.2 ‰ of the full estimate (owner's decision). The pass runs on AC power only (the timer's battery check; desktops count as AC); `ogc analyze --on-battery` overrides it. `estimate::GENERATION` is 2. The analysis cache keeps the estimator generation and size bucket. On Windows a measurement walk after a WOF run sums `GetCompressedFileSizeW` and checks every file's algorithm. **Contributions.** `collect.rs` applies §3.6 (store key and build, fully installed, complete run, no exclusions or skip list, Windows measured only after a clean walk, not again within 180 days for methods already sent; excluded games are never shared). The outbox holds core values only, one item per game; the sender adds salt, PoW and padding, POSTs exactly 1024 bytes over a fresh ureq agent (no keep-alive, no TLS resumption, no cookies, constant User-Agent, SOCKS resolving remotely), random not-before 1–24 h, ≥ 2 h apart, ≤ 8 a day, only while the app is open or on `ogc db send`; the timer only queues. 202 accepted; every 4xx deletes; 410 turns sharing off; 429/503 back off; 14-day expiry; a 30-day pause after failures on at least 3 days. **UI.** "Share results anonymously" switch with the §9.3 confirmation dialog, the one-time card after the first analysis, the outbox page (exact JSON, remove one, remove all), notices. CLI: `ogc db outbox [--show] [--clear]`, `ogc db send`, `ogc db measure` (curated measurement in ogc-db's format, no paths or names; on Windows real WOF runs with `--restore`). Review: 13 findings from two independent reviews (10 distinct, 1 high: Windows estimates and measurements replaced each other in the outbox), all fixed. Locally: fmt, clippy (both feature sets), 1850 tests incl. GUI under Broadway pass. Not run: the Windows GTK clippy (`check-gui.sh`) — CI covers it. Fixed after the first CI run: the thread id comes from the system call instead of `gettid()`, which glibc 2.28 (the bundle's floor) lacks.

SkyfaR added 2 commits 2026-10-07 07:43:59 +02:00

Community database in the app: shipped data, predictions, ogc db, opt-in download ... de8a0b1985

The app ships a snapshot of the community database and shows its expected
savings on game cards until the local analysis replaces them; predictions
never drive compressing, skipping, the timer or the dashboard's total.
ogc db status, update and show, prediction fields in ogc list's JSON and a
community line in ogc analyze. Downloads are off until switched on in the
settings, fetch whole signed files at most once a day, and stay disabled
until the release keys are embedded.

Share results anonymously: collection, outbox, sender, consent, ogc db measure ... f4372be259

With sharing switched on, an analysis also estimates the zstd anchors (and
XPRESS on Windows) in a separate pass at background priority, and a complete
analysis or a WOF measurement is queued per game under the design's rules.
The outbox sends one 1024-byte record at a time with proof of work over a
fresh connection, at random times, at most eight a day and only while the
app is open or on ogc db send; the background timer only queues. Sharing is
off until confirmed in a dialog; the queue can be inspected and cleared.
ogc db measure writes a curated measurement for ogc-db without paths or
names.

SkyfaR added 6 commits 2026-10-07 10:31:12 +02:00

Estimate zstd levels 9 and 15 from a subset of the samples ... bb2b341c33

Where a folder has at least 512 samples, levels 9 and 15 compress every
8th sample of a file read whole and every 2nd of a sampled one, at a
phase taken from the file's path, and scale what level 6 saves by the
folder's ratio of savings on those subsets. On Proton Experimental and
Steam's runtimes this cuts the CPU time of the pass that estimates every
method by 68 to 80 %, with level 15 within 2.2 per mille of the estimate
from every sample. Curated measurements still use every sample.

The estimator generation is 2.

Estimate every method for sharing on AC power only ... 2dc707e72d

The app's worker leaves the games noted while the computer runs on
battery, gives way when it is unplugged mid-pass, and resumes once a
thread finds it on AC power again. ogc analyze skips its pass on
battery with a note, unless --on-battery asks for it. The power check
is the timer's, so a computer without a battery counts as on AC power.

Estimate levels 9 and 15 the same whatever level is analysed ... 2c59216905

Where the analysis compresses at level 9 or 15 itself, its compressor
still reads every sample for the card's estimate, but the level's
estimate for the methods now comes from its subset like any other's.
The same folder then contributes the same values at any level analysed,
and level 15's ratio is bounded by level 9's before the payload's clamp.

Take every 4th sample of a sampled file for levels 9 and 15 ... 633c67567c

A game made of large, sampled files kept about 56 % of the CPU time of
the methods pass with every 2nd sample. With every 4th it keeps about a
third: a twice-copied Unity game went from 47.6 to 14.9 CPU s (-69 %),
Steam's runtimes and Proton Experimental by 69 to 82 %, about 8 to 11
times a plain analysis whatever the mix of files. Level 15 stays within
2.4 per mille of the estimate from every sample.

The docs give the cost by file mix, mark the older cost figures as
measured before the subset, and say that per extension levels 9 and 15
are projections of level 6's savings.

Merge main into contrib 5aeea0fac6

Take the thread id from the system call, not gettid() ... 51de06b8e2

glibc has gettid() only since 2.30; the bundle is built for 2.28, where
linking ogc failed with an undefined gettid.

SkyfaR referenced this pull request from a commit 2026-10-07 10:58:24 +02:00

Merge pull request 'Share results anonymously: collection, outbox, sender, consent, ogc db measure' (#48) from contrib into main

SkyfaR manually merged commit eb3ca57d53 into main 2026-10-07 10:58:26 +02:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

LevelXStudios/OpenGameCompressor!48

No description provided.