ogc-db

The community database of OpenGameCompressor: how much disk space installed games save with btrfs zstd on Linux and with NTFS (WOF) compression on Windows, aggregated from anonymous results and curated measurements.

This repository builds db.opengamecompressor.de: a browsable, ProtonDB-style site (search, one page per game with a tier per operating system, browse lists, statistics, methodology and privacy) and, on the same host, the static /v1/ data files that the app downloads.

Status: skeleton. The data model, the site generator and synthetic test data exist; the aggregator, the intake service and the deployment files follow (see the build plan).

Layout

Path	Content
model/	Rust types of the published /v1 files, consistency checks, the fixture markers
model/schema/	JSON Schemas of every published file (they reject the fixture marker)
model/examples/gen_fixtures/	Writes the synthetic test data under testdata/
site/	Generator of the static site (minijinja templates, English at /, German at /de/)
catalog/	ogc-db-catalog: game names from public store catalogues, self-hosted Steam header images
cache/	Catalog output for published games only: names.json (read by the aggregator), names-fetched.json, images.json and img/steam/ (read by the site); aggregate-state.json, what the anomaly gate already approved
rules/names.toml	Name overrides; the only source of Epic and Amazon names
rules/verdict.toml	Tier thresholds and the publication gate
testdata/v1/, testdata/v1-empty/	Synthetic data: 30 invented games, and an empty database
testdata/catalog/	Synthetic store answers and contributions for the catalog tests
public/	Build output for publishing (committed by the publish workflow, never fixture data)
.forgejo/workflows/, scripts/publish.sh	CI for every change, and the daily publish run (see Publishing)
LICENSE, LICENSE-DATA	Code: GPL-3.0-or-later. Published data: CC0-1.0

Building

Requires a Rust toolchain (edition 2024). Nothing else: no Node, no CDN.

cargo test                       # model, schemas, fixtures, generator
cargo run -p ogc-db-site -- build --data <v1 dir> --out public
cargo run -p ogc-db-site -- check-clean public
cargo run --bin ogc-db-aggregate -- check public/v1   # schemas, snapshot, k/week gate, markers

build validates the data (schemas' rules, the publication gate, that every tier matches rules/verdict.toml), writes all pages and copies the data unchanged to <out>/v1/. When the aggregator has already written public/v1/, --data public/v1 --out public builds the site around it and leaves v1/ (and a .gitkeep) untouched; any other overlap of data and output is refused. Options: --rules, --config (default site/site.toml), --images cache/images.json (the catalog's image manifest) and --image-root <dir> (where its img/ lives; default: the manifest's directory, which must lie outside the output). Every build rewrites <out>/img/ with the copies of the games in the data and nothing else.

Preview with test data

cargo run -p ogc-db-site -- build --data testdata/v1 --out build/preview --fixtures
python3 -m http.server --directory build/preview 8000

The generator refuses fixture data without --fixtures, and every page of such a build carries a banner. check-clean fails if a fixture marker ("fixture", an id starting with 999000, a name starting with "Fixture Game") appears anywhere in a directory; run it on public/ before publishing. Regenerate the fixtures with cargo run -p ogc-db-model --example gen_fixtures -- testdata.

Catalog: names and header images

cargo run -p ogc-db-catalog -- --index public/v1/index.json

The catalog reads the game keys of the published index only, never raw contributions: its output is committed to this public repository, so a game held back by the publication gate must not appear in it. CI runs it on the committed index before the aggregator, so a newly published game gets its name and image on the next run. Cache entries and copies of games that left the index are deleted. The catalog then asks the stores only about games whose cached answer is missing or old:

• Steam: store.steampowered.com/api/appdetails?appids=<id>, one app per request, for the name and the header_image URL. The old keyless full list (ISteamApps/GetAppList/v2) answered 404 when checked on 2026-10-06; its successor IStoreService/GetAppList needs a key.
• GOG: api.gog.com/products/<id> for the title.
• Epic, Amazon: no public source is used; names come from rules/names.toml, which also overrides any fetched name.

Steam header images are downloaded once, re-encoded (metadata dropped) as JPEG at 230 and 460 px (920 px only if the original is that wide) to cache/img/steam/<appid>-<width>.jpg, and listed in cache/images.json with source URL, ISO week and SHA-256. The site copies those of published games into its output and shows them with srcset, fixed dimensions and lazy loading in lists; games without one keep the fallback.

Every request sends User-Agent: OpenGameCompressor-DB (+https://db.opengamecompressor.de), goes only to an allowlist of store hosts, waits at least one second after the previous one, backs off on 429 and counts against a budget per run (--max-requests, default 600); the rest follows on the next run. Retry-After (seconds or an HTTP date) is honoured; a wait over 10 minutes, or one that cannot be read, ends the run instead of asking early. Names are checked again after 26 weeks, images after 13; unknown games, refused requests (another 4xx, an answer over the size limit, an unreadable answer) and missing images after 4. A later answer without a name keeps the known name. --offline only rebuilds the caches from the index and the overrides.

Publishing

CI (.forgejo/workflows/ci.yml, every pull request and push to main): cargo fmt --check, clippy with -D warnings, cargo test --workspace, check-clean and ogc-db-aggregate check public/v1 on the committed output (every file against its schema, the snapshot against the manifest, the k/week gate, no fixture marker), and a dry run of the publish steps on curated/ without raw data and without signing.

Publish (.forgejo/workflows/publish.yml, daily at 04:23 UTC and by hand under Actions → Publish, on main only), one step of scripts/publish.sh each:

1. Raw contributions: a shallow checkout of LevelXStudios/ogc-db-raw with OGC_DB_RAW_TOKEN; without it, curated measurements only.
2. The catalog on the committed public/v1/index.json (skipped while there is none).
3. The aggregator into build/, which decides the anomaly gate; the catalog on that run's index, so newly published games get their names; the aggregator again, with the names, into public/v1/. Both passes read the same previous tree, state and time.
4. The site around public/v1/, then the gates: no fixture marker, ogc-db-aggregate check.
5. minisign -S -t "serial=<n>" on latest.json, verified with MINISIGN_PUBLIC_KEY.
6. One commit of public/ and cache/ (never report.md or the aggregator's .v1.ogc-new/ .v1.ogc-old trees), pushed to main with OGC_DB_PUSH_TOKEN. Its message ends in [skip ci], so CI does not run on it again, and nothing runs publish.yml on a push, so nothing loops. The push fires the deploy webhook (deploy/README.md), which puts the new public/ online.

When the anomaly gate trips, the run fails with "the anomaly gate stopped the run" and nothing is committed. Download the artifact report-md-age of that run, decrypt it (age -d -i <identity file> report.md.age, or -i ~/.ssh/id_ed25519 for an SSH recipient), read it, and if the change is legitimate run Publish by hand with approve ticked. Otherwise add an exclusion to rules/exclude.toml. The report names published games with their weekly record counts and the raw file names, so it is meant for the maintainer only. Forgejo lets anyone who can see a public repository download its Actions artifacts without logging in (checked on forgejo.skyfar.de, Forgejo 15.0.9: an anonymous request for an artifact of the public main repository was answered with the file). The report is therefore only kept encrypted, and not at all without OGC_REPORT_RECIPIENTS. The job log never prints it.

Secrets (repository settings → Actions → Secrets):

Secret	Needed	Content
OGC_SALT_SECRET	yes	The content of /etc/ogc-intake/salt_secret on the VM, without the trailing line break
OGC_DB_PUSH_TOKEN	yes	Access token (scope write:repository) of a bot account that may push to main; if main is protected, allow that account to push. Publish commits are not GPG-signed
MINISIGN_SECRET_KEY	yes, unless unsigned publishing is allowed	The content of the CI key's secret key file
MINISIGN_PASSWORD	if the key has one	The CI key's password
OGC_DB_RAW_TOKEN	for crowd data	Token (scope read:repository) of an account that can read LevelXStudios/ogc-db-raw, and nothing more

Variables (repository settings → Actions → Variables):

Variable	Content
MINISIGN_PUBLIC_KEY	The CI key's public key (the RW… line of its .pub file); every new signature is verified with it before the commit. Recommended; without it each signed run warns that nothing was verified
OGC_REPORT_RECIPIENTS	Who can read report.md: age1… public keys (age-keygen) or SSH public keys (ssh-ed25519 …), one per line. Recommended
OGC_DB_ALLOW_UNSIGNED	true publishes an unsigned latest.json while MINISIGN_SECRET_KEY is not set, with a warning in every such run. The app refuses unsigned data, so this only helps before the first release that reads it. Once main carries a latest.json.minisig, a run without the key fails whatever this says; to go back to unsigned on purpose, delete that file on main in a commit
OGC_DB_SUBMIT, OGC_DB_POW_BITS, OGC_DB_MIN_GEN	The manifest's submit, pow_bits and min_gen; empty keeps the published values (at first false, 22 and 1). pow_bits must equal POW_BITS in the VM's switches.env
OGC_DB_RAW_URL	Clone URL of the raw repository, if not <this Forgejo>/LevelXStudios/ogc-db-raw.git
OGC_DB_BOT_NAME, OGC_DB_BOT_EMAIL	Author of the publish commits (default ogc-db publish)

The minisign keys (design §5.5), made on the owner's machine:

minisign -G -p ogc-db-ci.pub -s ogc-db-ci.key            # CI key; its password goes into MINISIGN_PASSWORD
minisign -G -p ogc-db-backup.pub -s ogc-db-backup.key    # backup key: kept offline, never in CI

The content of ogc-db-ci.key goes into MINISIGN_SECRET_KEY, the second line of ogc-db-ci.pub into MINISIGN_PUBLIC_KEY; both public keys go into the app. minisign -G -W makes a key without a password.

Privacy of the site

The pages load nothing from other hosts: fonts (Chakra Petch, IBM Plex, SIL OFL 1.1) and all scripts are served from the site itself. JavaScript is only used for search and filters; every page works without it.