Community database in the app: shipped data, predictions, ogc db, opt-in download #47

Manually merged

SkyfaR merged 2 commits from communitydb into main 2026-10-07 09:18:51 +02:00

Conversation 0 Commits 2 Files changed 4 +14 -4

SkyfaR commented 2026-10-07 04:01:50 +02:00

Owner

Copy link

Phases 1 and 2 of the community database design, on top of #43 (ogc-db-proto; this branch contains its commit, so merge #43 first).

Data. src/communitydb/ reads the published snapshot (format of LevelXStudios/ogc-db's model), checks the minisign signature of latest.json against embedded primary/backup keys, the snapshot's length and SHA-256, anti-rollback by serial (a floor that survives a discarded cache), and marks expired data stale. The app ships data/communitydb/ (serial 1: the nine curated Windows/LZX entries from ogc-db#1); scripts/communitydb-snapshot.sh refreshes it. Db::load takes the newer of shipped and cached data.

Lookup. Game::portable_key() for Steam, Epic, GOG and Amazon (Heroic too); None for everything without a store ID. Variant by content platform and size class.

App. Cards show the expected saving with "community data" and confidence until the local analysis replaces it; estimates are requested biggest predicted saving first. Predictions never drive Need::Small, auto-compress, the timer, skip decisions or the dashboard's "worth" total (separate line). Recommendation, poorly compressing types, curated notes (asked before compressing). Settings group "Community database" with the download switch (off by default), status and privacy link.

CLI. ogc db status, ogc db update (asking is consent), ogc db show <game|appid|key> [--json]; ogc list --format=json gains predicted_savings_bytes, prediction_confidence, prediction_source, prediction_method; ogc analyze adds a community line.

Download. ureq 3 + rustls (webpki-roots), whole files only, new agent per source, constant User-Agent, proxies honoured, size and time limits, at most once a day, Forgejo raw mirror after a failed primary. Disabled until the release keys exist (data/communitydb/keys/{primary,backup}.pub; the switch is greyed out with an explanation). No sending yet (phase 3).

Packaging: the Arch package's source tarball is built with tar, since cargo package cannot take the path dependency.

Review: 11 findings from two independent reviews, all handled. Locally: fmt, clippy (both feature sets), all tests incl. 630 GUI tests under Broadway pass. Translations beyond English and German are my own and need a native speaker's look.

Phases 1 and 2 of the community database design, on top of #43 (`ogc-db-proto`; this branch contains its commit, so merge #43 first). **Data.** `src/communitydb/` reads the published snapshot (format of `LevelXStudios/ogc-db`'s model), checks the minisign signature of `latest.json` against embedded primary/backup keys, the snapshot's length and SHA-256, anti-rollback by serial (a floor that survives a discarded cache), and marks expired data stale. The app ships `data/communitydb/` (serial 1: the nine curated Windows/LZX entries from ogc-db#1); `scripts/communitydb-snapshot.sh` refreshes it. `Db::load` takes the newer of shipped and cached data. **Lookup.** `Game::portable_key()` for Steam, Epic, GOG and Amazon (Heroic too); `None` for everything without a store ID. Variant by content platform and size class. **App.** Cards show the expected saving with "community data" and confidence until the local analysis replaces it; estimates are requested biggest predicted saving first. Predictions never drive `Need::Small`, auto-compress, the timer, skip decisions or the dashboard's "worth" total (separate line). Recommendation, poorly compressing types, curated notes (asked before compressing). Settings group "Community database" with the download switch (off by default), status and privacy link. **CLI.** `ogc db status`, `ogc db update` (asking is consent), `ogc db show <game|appid|key> [--json]`; `ogc list --format=json` gains `predicted_savings_bytes`, `prediction_confidence`, `prediction_source`, `prediction_method`; `ogc analyze` adds a community line. **Download.** ureq 3 + rustls (webpki-roots), whole files only, new agent per source, constant User-Agent, proxies honoured, size and time limits, at most once a day, Forgejo raw mirror after a failed primary. **Disabled until the release keys exist** (`data/communitydb/keys/{primary,backup}.pub`; the switch is greyed out with an explanation). No sending yet (phase 3). Packaging: the Arch package's source tarball is built with `tar`, since `cargo package` cannot take the path dependency. Review: 11 findings from two independent reviews, all handled. Locally: fmt, clippy (both feature sets), all tests incl. 630 GUI tests under Broadway pass. Translations beyond English and German are my own and need a native speaker's look.

SkyfaR added 2 commits 2026-10-07 04:01:50 +02:00

Add ogc-db-proto, the community database's contribution protocol ... c936287435

A workspace member with the payload of the community database: game and
method keys, coarse values, the canonical 1024-byte body, the proof of work
with its weekly salt, and the validator the intake service will share.
Nothing in the app uses it yet. The workspace excludes target/, where the
Arch package and GTK's build unpack crates of their own.

Community database in the app: shipped data, predictions, ogc db, opt-in download ... de8a0b1985

The app ships a snapshot of the community database and shows its expected
savings on game cards until the local analysis replaces them; predictions
never drive compressing, skipping, the timer or the dashboard's total.
ogc db status, update and show, prediction fields in ogc list's JSON and a
community line in ogc analyze. Downloads are off until switched on in the
settings, fetch whole signed files at most once a day, and stay disabled
until the release keys are embedded.

SkyfaR referenced this pull request 2026-10-07 07:43:59 +02:00

Share results anonymously: collection, outbox, sender, consent, ogc db measure #48

SkyfaR added 1 commit 2026-10-07 08:20:11 +02:00

Embed the community database's release keys ... 027d9a8cdb

primary.pub verifies what ogc-db's publish run signs; backup.pub is the
offline key for a rotation. With them the download switch is available.

SkyfaR referenced this pull request 2026-10-07 08:20:27 +02:00

VM test: count a game compressed as ogc does, not by every extent #49

SkyfaR added 1 commit 2026-10-07 08:55:30 +02:00

Windows check: allow ws2_32.dll and bcrypt.dll for the download ... caeaa4f069

With the release keys embedded, ogc.exe links the download: sockets from
ws2_32.dll and, through ring's getrandom, BCryptGenRandom from bcrypt.dll.
Neither is known to load user32.dll or gdi32.dll; a real-PC check of the
running program's modules is still to be done.

SkyfaR referenced this pull request from a commit 2026-10-07 08:55:47 +02:00

Merge pull request 'Community database in the app: shipped data, predictions, ogc db, opt-in download' (#47) from communitydb into main

SkyfaR referenced this pull request from a commit 2026-10-07 09:18:49 +02:00

Merge pull request 'Community database in the app: shipped data, predictions, ogc db, opt-in download' (#47) from communitydb into main (release keys, Windows imports)

SkyfaR manually merged commit 1e80f5e6aa into main 2026-10-07 09:18:51 +02:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

LevelXStudios/OpenGameCompressor!47

No description provided.