Share identical files, and compress Proton prefixes #8

Merged

SkyfaR merged 3 commits from dedupe-and-prefixes into main 2026-10-01 09:23:24 +02:00

Conversation 0 Commits 3 Files changed 59 +2244 -137

SkyfaR commented 2026-10-01 09:07:26 +02:00

Owner

Copy link

Two ideas from the roadmap: ogc dedupe, and compressing Proton prefixes. Command line and timer only; the desktop
app gets them later.

ogc dedupe

Lets identical files of games and Proton prefixes keep their data once on btrfs, like duperemove on whole files.

• Only the kernel decides that two files are identical: FIDEDUPERANGE compares them byte by byte with both locked.
  No file is written; markers, times and modes stay (checked on a real kernel). The owner may share into read-only
  files without write permission.
• Only files the compressor has finished with take part, since a later rewrite would give a shared file its own copy
  again. Identical files that are not compressed yet are reported with the advice to compress them first.
• Files that share already are found by their extent map and not read again, so repeated runs are cheap. The most
  compressed copy becomes the source; a copy whose data a snapshot or a file outside the selection keeps anyway is
  preferred, and is never used as a destination (that would free nothing). A copy that agrees with the source only
  at the start is tried against the others.
• ENOSPC/EROFS end the run, other errors (a checksum error, an immutable or swap file) skip one file; a filesystem
  with less than 256 MiB free is left out. Needs Linux 5.10, refuses alongside bees or btdu. --dry-run compares in
  user space and changes nothing.

Behaviour change: a compression at another level, or with --force, now leaves files whose compressed data is
mostly shared as they are (without a marker), so it does not undo the sharing. --unshare rewrites them anyway.
Shared data that is not compressed yet (snapshots of old installs, Proton's copies in its prefixes, converted
extents) is compressed as before.

Proton prefixes

--prefixes adds the prefix of each selected game (steamapps/compatdata/<appid>, once Proton has set it up) to
compress, analyze and dedupe. A prefix is in use while a process has it in STEAM_COMPAT_DATA_PATH, which also
catches wineserver; ogc auto waits while any such process runs, including games from outside Steam. Compressed
prefixes are recorded in prefixes.tsv with a stamp of the files Wine rewrites when the prefix is used.

Timer

ogc timer enable --include-prefixes --dedupe: changed prefixes are compressed, and identical files are shared after a
compression or once a week (a failed run counts too, so it is not repeated hourly). Both are off unless chosen; the
desktop app keeps them when it rewrites the unit.

Checked

• formatting, clippy with and without the desktop app, all tests
• new btrfs tests on a real btrfs filesystem (kernel 7.2): identical compressed files end up sharing their extents,
  a read-only destination works without write permission, nothing changes in a dry run, files that are not compressed
  yet are left out, a new level keeps shared files until their copy is gone, sharing after a partial match, and
  sharing onto data that something outside the selection keeps; each safeguard was removed once and a test failed
• a new VM scenario space (prefix compressed and recorded, dry run, sharing confirmed by btrfs filesystem du, a
  new level keeps the shared copies, the timer, btrfs scrub, remount, checksums); all six scenarios pass with KVM
• by hand: ogc compress --prefixes, ogc dedupe and ogc auto --include-prefixes --dedupe on a made-up library in
  a throwaway home folder
• an adversarial review of the change; its findings are fixed (among them a too broad keep-shared rule that would have
  kept snapshotted and converted data uncompressed, a checksum error that stopped every timer run, a nearly full drive
  that stopped deduplication everywhere, prefixes of two installs of one game recompressed hourly, and the "game
  running" check missing Proton games from outside Steam)

Not checked here

How much this saves on a real library is not measured. ogc dedupe --all --prefixes --dry-run shows it without
changing anything. The DLLs Proton places in its prefixes share their data with the Proton installation until either
is compressed; afterwards ogc dedupe has to share the compressed copies again.

Two ideas from the roadmap: `ogc dedupe`, and compressing Proton prefixes. Command line and timer only; the desktop app gets them later. ## `ogc dedupe` Lets identical files of games and Proton prefixes keep their data once on btrfs, like duperemove on whole files. - Only the kernel decides that two files are identical: `FIDEDUPERANGE` compares them byte by byte with both locked. No file is written; markers, times and modes stay (checked on a real kernel). The owner may share into read-only files without write permission. - Only files the compressor has finished with take part, since a later rewrite would give a shared file its own copy again. Identical files that are not compressed yet are reported with the advice to compress them first. - Files that share already are found by their extent map and not read again, so repeated runs are cheap. The most compressed copy becomes the source; a copy whose data a snapshot or a file outside the selection keeps anyway is preferred, and is never used as a destination (that would free nothing). A copy that agrees with the source only at the start is tried against the others. - `ENOSPC`/`EROFS` end the run, other errors (a checksum error, an immutable or swap file) skip one file; a filesystem with less than 256 MiB free is left out. Needs Linux 5.10, refuses alongside bees or btdu. `--dry-run` compares in user space and changes nothing. **Behaviour change:** a compression at another level, or with `--force`, now leaves files whose *compressed* data is mostly shared as they are (without a marker), so it does not undo the sharing. `--unshare` rewrites them anyway. Shared data that is not compressed yet (snapshots of old installs, Proton's copies in its prefixes, converted extents) is compressed as before. ## Proton prefixes `--prefixes` adds the prefix of each selected game (`steamapps/compatdata/<appid>`, once Proton has set it up) to `compress`, `analyze` and `dedupe`. A prefix is in use while a process has it in `STEAM_COMPAT_DATA_PATH`, which also catches wineserver; `ogc auto` waits while any such process runs, including games from outside Steam. Compressed prefixes are recorded in `prefixes.tsv` with a stamp of the files Wine rewrites when the prefix is used. ## Timer `ogc timer enable --include-prefixes --dedupe`: changed prefixes are compressed, and identical files are shared after a compression or once a week (a failed run counts too, so it is not repeated hourly). Both are off unless chosen; the desktop app keeps them when it rewrites the unit. ## Checked - formatting, clippy with and without the desktop app, all tests - new btrfs tests on a real btrfs filesystem (kernel 7.2): identical compressed files end up sharing their extents, a read-only destination works without write permission, nothing changes in a dry run, files that are not compressed yet are left out, a new level keeps shared files until their copy is gone, sharing after a partial match, and sharing onto data that something outside the selection keeps; each safeguard was removed once and a test failed - a new VM scenario `space` (prefix compressed and recorded, dry run, sharing confirmed by `btrfs filesystem du`, a new level keeps the shared copies, the timer, `btrfs scrub`, remount, checksums); all six scenarios pass with KVM - by hand: `ogc compress --prefixes`, `ogc dedupe` and `ogc auto --include-prefixes --dedupe` on a made-up library in a throwaway home folder - an adversarial review of the change; its findings are fixed (among them a too broad keep-shared rule that would have kept snapshotted and converted data uncompressed, a checksum error that stopped every timer run, a nearly full drive that stopped deduplication everywhere, prefixes of two installs of one game recompressed hourly, and the "game running" check missing Proton games from outside Steam) ## Not checked here How much this saves on a real library is not measured. `ogc dedupe --all --prefixes --dry-run` shows it without changing anything. The DLLs Proton places in its prefixes share their data with the Proton installation until either is compressed; afterwards `ogc dedupe` has to share the compressed copies again.

SkyfaR added 3 commits 2026-10-01 09:07:26 +02:00

Share identical files, and compress Proton prefixes ... b646e682c9

ogc dedupe lets identical files of games and Proton prefixes keep their
data once on btrfs, as duperemove does with whole files. Only the
kernel decides that two files are identical: FIDEDUPERANGE compares
them byte by byte with both locked, and no file is written, so markers,
times and modes stay. Only files the compressor has finished with take
part, since a later rewrite would give a shared file its own copy
again. Files that share their data already are found by their extent
map and not compared again; the most compressed copy becomes the
source, a copy whose data a snapshot keeps anyway is preferred, and a
copy that matches only at the start is tried against the others. A
nearly full filesystem is left out; ENOSPC and EROFS end the run,
other errors skip one file. It needs Linux 5.10 and refuses to run
alongside bees or btdu; --dry-run compares in user space.

A compression at another level, or with --force, now leaves files
whose compressed data is mostly shared as they are, without a marker,
so that it does not undo the sharing; --unshare rewrites them anyway.
Shared data that is not compressed yet is compressed as before.

--prefixes adds the Proton prefix of each selected game (once Proton
has set it up) to compress, analyze and dedupe. A prefix is in use
while a process has it in STEAM_COMPAT_DATA_PATH; ogc auto waits while
any such process runs, also for games from outside Steam. Compressed
prefixes are recorded in prefixes.tsv with a stamp of the files Wine
rewrites when the prefix is used.

The timer gains --include-prefixes and --dedupe (deduplication at most
after a compression or once a week), both off unless chosen; the
desktop app keeps them when it rewrites the unit.

Test prefixes and shared files in the VM ... 814c768a96

The new scenario "space" works on a drive that is btrfs already: both
games and the prefix of the first hold the same runtime. It checks that
the prefix is compressed and recorded, that a dry run changes nothing,
that the copies share their data afterwards according to btrfs, that a
compression at another level keeps them shared, and the timer, a
scrub, a remount and the checksums. CI runs it with the other five.

Document deduplication and Proton prefixes 76b4d95feb

SkyfaR merged commit 34aedc29f0 into main 2026-10-01 09:23:24 +02:00

SkyfaR referenced this pull request 2026-10-06 05:06:13 +02:00

Windows port, part 4a: Epic, GOG, Ubisoft, EA and Heroic on Windows, and the dual-boot hints (W4 part 1) #35

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

LevelXStudios/OpenGameCompressor!8

No description provided.