Windows port, part 1: measurement probe (W0) and portability groundwork (W1) #29

Manually merged

SkyfaR merged 9 commits from windows-port into main 2026-10-04 15:24:09 +02:00

Conversation 0 Commits 9 Files changed 184 +16137 -950

SkyfaR commented 2026-10-04 11:20:22 +02:00

Owner

Copy link

Start of the Windows port: the measurement probe for a real Windows PC (phase W0) and the portability groundwork in the Linux code (phase W1). Linux behaviour and Linux file formats do not change.

W0: ogc-probe.exe (tools/ogc-probe/, a crate of its own outside the Linux build)

• A standalone Windows x64 program, cross-built in a container, 2 MB, statically linked. It needs no installation and no admin rights, has no network code, and uses a German menu.
• Reading commands:
  • umgebung: Windows version, volumes and WOF availability, Steam libraries (registry, libraryfolders.vdf), Epic/GOG/Ubisoft/EA games, write-access checks without writing, Defender and Controlled Folder Access.
  • bestand: how each game file is stored (WOF, LZNT1, reparse points, DirectStorage).
  • schaetzen / messen: estimates for XPRESS4K/8K/16K and LZX and for btrfs zstd 1/3/6/9/15. It takes samples, calibrated by really WOF-compressing a sample in its own temp folder.
  • selbsttest: works in a folder of its own and removes it.
• Optional one-game test (spiel / pruefen / zurueck), only after typing the game's key, with Steam closed:
  • It compresses only plain files. It skips hard links, sparse, encrypted, cloud and offline, read-only, LZNT1, other WOF types, reparse points and files in use.
  • A record is written atomically before anything changes. It hashes every file before and after Steam's "verify integrity".
  • It restores exactly, verifying every file before the record is deleted. A record that is in progress or damaged is never overwritten.
• Report (ogc-probe-bericht.json / .txt), in the curated measurement format of the database design. It contains no user, computer or domain names, no paths and no serials. It is checked before saving, shown, and saved only after confirmation. The console shows the profile folder as %USERPROFILE%.
• Robustness: Windows "no disk" dialogs are suppressed. Under Wine, the WOF functions Wine lacks are reported as "not available" instead of crashing.
• scripts/windows/measure-wof.ps1: a compact.exe fallback with the same privacy rules. It decompresses exactly the files it compressed, also after an error or Ctrl+C.
• German guide: tools/ogc-probe/ANLEITUNG.txt.
• Release: the private release probe-1 in LevelXStudios/ogc-messungen is built from this code.

W1: portability groundwork

• src/paths.rs: one place decides where files go. Linux keeps the exact XDG behaviour; Windows uses %LOCALAPPDATA% and %APPDATA%. It replaces seven copies of the layout.
• Codec, read_at, file locking and WriteStop:
  • The codec abstraction keeps btrfs zstd levels and WOF algorithms in one place.
  • read_at and file locking (File::lock) are portable.
  • WriteStop is classified by io::ErrorKind.
• Linux-only parts: src/platform/linux/ (running, power, notify, re-exported under their old names), and cfg gates for the migration, the helper, systemd, btrfs ioctls, polkit, udisks and the procfs detection.
• Windows stubs: they say clearly "not supported yet". ogc list on Windows says so and exits 1 instead of "found nothing".
• State files stay byte-identical: a new test (tests/state_files.rs) writes and reads every state file and compares it to the old format.
• CI: two new jobs.
  • windows-check: clippy of the library, ogc and their tests for x86_64-pc-windows-gnullvm, with llvm-mingw, the toolchain and UCRT the Windows release will use.
  • windows-wine-test: the portable unit tests under Wine, in a prefix of their own and without a display.
• Local: scripts/windows/in-container.sh runs the same jobs in a container. The host needs neither the target nor llvm-mingw nor Wine.

Review

• Process: both parts were reviewed independently and every finding was verified adversarially. 18 were confirmed, all fixed. An independent check of the probe rated it "ready to hand over"; its three remaining minor risks are closed as well.
• Main probe fixes:
  • The restore now verifies before it deletes its record.
  • Special files are skipped.
  • An interrupted run never loses the original state.
  • The PowerShell fallback restores exactly the files it compressed.
  • The estimate shows no savings for incompressible data.
• Main groundwork fixes:
  • The W1 acceptance is met: Wine tests and clippy over the test code.
  • The gnullvm triple is used, as planned.

Checks

• cargo fmt, both clippy runs, and all tests with and without the GUI (Broadway) are green on Linux.
• All seven VM scenarios pass on this branch, including rollback with five boots.
• Windows target:
  • Clippy is clean.
  • Under Wine: library 174 passed, ogc 25 passed.
  • Probe: 80 tests natively, 93 under Wine, the Wine smoke test ("report checks: ok"), and the PowerShell tests.

Not verified: every real WOF call. Wine has no WOF, so they ran against fakes. That is what the probe on the friend's PC is for.

Start of the Windows port: the measurement probe for a real Windows PC (phase W0) and the portability groundwork in the Linux code (phase W1). Linux behaviour and Linux file formats do not change. ## W0: `ogc-probe.exe` (`tools/ogc-probe/`, a crate of its own outside the Linux build) - A standalone Windows x64 program, cross-built in a container, 2 MB, statically linked. It needs no installation and no admin rights, has no network code, and uses a German menu. - **Reading commands:** - `umgebung`: Windows version, volumes and WOF availability, Steam libraries (registry, `libraryfolders.vdf`), Epic/GOG/Ubisoft/EA games, write-access checks without writing, Defender and Controlled Folder Access. - `bestand`: how each game file is stored (WOF, LZNT1, reparse points, DirectStorage). - `schaetzen` / `messen`: estimates for XPRESS4K/8K/16K and LZX and for btrfs zstd 1/3/6/9/15. It takes samples, calibrated by really WOF-compressing a sample in its own temp folder. - `selbsttest`: works in a folder of its own and removes it. - **Optional one-game test** (`spiel` / `pruefen` / `zurueck`), only after typing the game's key, with Steam closed: - It compresses only plain files. It skips hard links, sparse, encrypted, cloud and offline, read-only, LZNT1, other WOF types, reparse points and files in use. - A record is written atomically before anything changes. It hashes every file before and after Steam's "verify integrity". - It restores exactly, verifying every file before the record is deleted. A record that is in progress or damaged is never overwritten. - **Report** (`ogc-probe-bericht.json` / `.txt`), in the curated measurement format of the database design. It contains no user, computer or domain names, no paths and no serials. It is checked before saving, shown, and saved only after confirmation. The console shows the profile folder as `%USERPROFILE%`. - **Robustness:** Windows "no disk" dialogs are suppressed. Under Wine, the WOF functions Wine lacks are reported as "not available" instead of crashing. - `scripts/windows/measure-wof.ps1`: a `compact.exe` fallback with the same privacy rules. It decompresses exactly the files it compressed, also after an error or Ctrl+C. - **German guide:** `tools/ogc-probe/ANLEITUNG.txt`. - **Release:** the private release `probe-1` in `LevelXStudios/ogc-messungen` is built from this code. ## W1: portability groundwork - **`src/paths.rs`:** one place decides where files go. Linux keeps the exact XDG behaviour; Windows uses `%LOCALAPPDATA%` and `%APPDATA%`. It replaces seven copies of the layout. - **Codec, `read_at`, file locking and `WriteStop`:** - The codec abstraction keeps btrfs zstd levels and WOF algorithms in one place. - `read_at` and file locking (`File::lock`) are portable. - `WriteStop` is classified by `io::ErrorKind`. - **Linux-only parts:** `src/platform/linux/` (`running`, `power`, `notify`, re-exported under their old names), and `cfg` gates for the migration, the helper, systemd, btrfs ioctls, polkit, udisks and the procfs detection. - **Windows stubs:** they say clearly "not supported yet". `ogc list` on Windows says so and exits 1 instead of "found nothing". - **State files stay byte-identical:** a new test (`tests/state_files.rs`) writes and reads every state file and compares it to the old format. - **CI:** two new jobs. - `windows-check`: clippy of the library, `ogc` and their tests for `x86_64-pc-windows-gnullvm`, with llvm-mingw, the toolchain and UCRT the Windows release will use. - `windows-wine-test`: the portable unit tests under Wine, in a prefix of their own and without a display. - **Local:** `scripts/windows/in-container.sh` runs the same jobs in a container. The host needs neither the target nor llvm-mingw nor Wine. ## Review - **Process:** both parts were reviewed independently and every finding was verified adversarially. 18 were confirmed, all fixed. An independent check of the probe rated it "ready to hand over"; its three remaining minor risks are closed as well. - **Main probe fixes:** - The restore now verifies before it deletes its record. - Special files are skipped. - An interrupted run never loses the original state. - The PowerShell fallback restores exactly the files it compressed. - The estimate shows no savings for incompressible data. - **Main groundwork fixes:** - The W1 acceptance is met: Wine tests and clippy over the test code. - The gnullvm triple is used, as planned. ## Checks - `cargo fmt`, both clippy runs, and all tests with and without the GUI (Broadway) are green on Linux. - All seven VM scenarios pass on this branch, including `rollback` with five boots. - **Windows target:** - Clippy is clean. - Under Wine: library 174 passed, `ogc` 25 passed. - Probe: 80 tests natively, 93 under Wine, the Wine smoke test ("report checks: ok"), and the PowerShell tests. **Not verified:** every real WOF call. Wine has no WOF, so they ran against fakes. That is what the probe on the friend's PC is for.

SkyfaR added 6 commits 2026-10-04 11:20:22 +02:00

Lay the groundwork for the Windows port without changing Linux ... c932382b4a

Phase W1 of the Windows plan, as far as it goes here: the code draws the
boundaries between what is shared and what each system does its own way,
and the library and ogc type-check for Windows. Linux behaviour and Linux
file formats stay as they were.

- paths: one module for where the program keeps its files, replacing the
  XDG copies in state, exclude, the language preference, the GUI settings,
  the migration backup, the launchers' folders and the artwork cache. XDG
  on Linux exactly as before; %LOCALAPPDATA% and %APPDATA% on Windows.
- codec: Codec::Zstd(level) and Codec::Wof(WofAlgo), with the text forms
  zstd:3 and wof:xpress8k. ZSTD_LEVELS is defined only there. games.tsv,
  prefixes.tsv and analysis.tsv read the level in both forms; Linux still
  writes it bare. A test keeps Microsoft's WOF numbers out of every file
  but the future Windows backend.
- platform: linux.rs and windows.rs with the same functions (positional
  reads, file identity, local time, private folders, mount roots, the
  terminal's screen reader, the folder layout); cfg appears only there and
  where modules are declared.
- Locks use File::lock, which is flock on Linux as before. WriteStop tells
  a full, read-only or over-quota filesystem by io::ErrorKind.
- The engine, the migration, the helper, systemd, the /proc checks, power,
  notifications and signals are declared for Linux only. On Windows,
  stand-ins of the same API say "This works only on Linux so far" in all
  13 languages; ogc list, exclude, history, completions and manpages work.
  ogc-migrate-helper builds everywhere and exits with 2 off Linux.
- Tests: tests/state_files.rs writes every state and settings file of the
  library in a home of its own and compares it byte for byte with the old
  format (it passes unchanged against the previous commit too); the
  analysis cache gets the same check. btrfs, migration and Linux-only
  unit tests are gated, so the test targets compile for Windows.
- CI: the job windows-check runs cargo check and clippy for
  x86_64-pc-windows-gnu without the desktop app in rust:1-trixie with
  mingw-w64.

Add ogc-probe, the Windows measurement probe for the port (phase W0) ... 551b262ffe

ogc-probe.exe answers the port plan's open questions H1-H21 on a real
Windows PC and takes curated WOF measurements for the database design
(section 10). It is a crate of its own in tools/ogc-probe with its own
lock file, so the Linux build and tests do not see it.

- umgebung, bestand, schaetzen, messen: volumes, Steam and other
  launchers, effective access, Defender, the files' current storage, and
  the sampling estimator for XPRESS4K/8K/16K, LZX and btrfs zstd,
  calibrated by compressing a file of the samples with real WOF.
- selbsttest: WOF on the probe's own test files in a folder it removes
  afterwards (access masks, size APIs, times, special files, not
  beneficial, readers and writers, large files, cancel, kill, speed).
- spiel, pruefen, zurueck: compress one game after confirmation with
  Steam closed, check it after Steam's verify, and restore it.
- laeuft: Steam's running values and the notification state.
- The report holds only codes, numbers and game keys; a check refuses
  paths, names and the user and computer names, and the friend sees the
  new content before it is saved.

scripts/windows/measure-wof.ps1 is the compact.exe fallback with the same
privacy rules. The exe is cross-built in a container; its tests run on
Linux, under Wine and with PowerShell 7.

Finish the Windows groundwork: gnullvm checks, unit tests under Wine, platform/linux ... d9d0059660

- CI: windows-check now runs clippy with all targets, so the tests' Windows code is checked
  too, and the new job windows-wine-test runs the unit tests of the library and ogc under Wine,
  with a prefix of its own and no crash dialog, then starts ogc.exe. Both build for
  x86_64-pc-windows-gnullvm with llvm-mingw pinned by SHA-256, the toolchain and CRT the
  Windows release is built with (plan, section 1.5 and D6), instead of -gnu. The scripts in
  scripts/windows/ run the same locally in a container.
- Test fixtures made path-neutral: history details and the list's home folder build absolute
  paths with paths::abs, Heroic and Faugus fixtures escape paths for JSON. The save of
  excluded.tsv that overlaps without the lock is tested on Linux only, as Windows may refuse
  to rename over a file another rename replaces.
- ogc list on Windows lists only own folders, says on stderr in every language that Steam and
  the other launchers are not looked for there yet, and fails when there is nothing to list.
  Linux is unchanged (platform::FINDS_LAUNCHERS).
- folders.tsv takes its folder from paths, as the other settings files do.
- The processes, power and notifications code moved to platform/linux/, re-exported at the
  crate root under the old names.

Make the probe's one-game test restore exactly, and fix the review's probe findings ... d7c2226e89

The probe runs on a friend's gaming PC, so restoring must be exact and
checked before anything that could undo a change is thrown away.

- zurueck touches only files stored differently from the record, then reads
  every file again; the state file is removed only when every file the test
  may touch is stored as before and no call failed. Otherwise it says how
  many differ and why, and the record stays for another try. LZNT1, other
  WOF providers and unreadable states are left alone (probe-1).
- spiel leaves alone, with the reason recorded: hard links, sparse,
  encrypted, cloud and offline files, LZNT1, other WOF providers, foreign
  reparse points and files another program holds open. Cloud and offline
  files are neither hashed nor sampled (probe-2).
- The state file is written through a temp file and a rename with
  kept=in_progress before anything changes; spiel refuses while any record
  exists, and a damaged one is told apart from none (probe-3).
- measure-wof.ps1 notes each file's storage first, compresses only plain
  ordinary files by name, keeps files compressed before (CompactGUI) as they
  are, writes the list before compressing, decompresses exactly those in a
  finally block, checks every file (restored_exactly), and offers to finish
  an interrupted run on the next start (probe-4).
- The calibration scales the modelled saving, so incompressible files stay
  at their size (probe-6), and puts samples into the calibration file by the
  bytes they stand for, not by file count (probe-7).
- A full disk or a cancel stops starting new files (probe-8); with all four
  algorithms, each run starts from the original state (probe-9).
- FSCTL_GET_EXTERNAL_BACKING gets room for the WIM provider, and
  ERROR_MORE_DATA counts as another provider (probe-10).
- A bare number is always an app id; list numbers need '#'. zurueck without
  a record decompresses only after the key is typed (probe-11).
- Game keys and builds that contain a short user name become a neutral code
  instead of blocking the report; the refusal names the field (probe-12).
- The Steam root is the first registry value whose folder exists, and
  config\libraryfolders.vdf is read when steamapps has none (probe-13).
- SetErrorMode suppresses "no disk in drive" boxes (probe-14).
- The guide says what restoring can and cannot promise, warns against '3'
  after an interrupted run, and names the fallback's report (probe-15).

Console paths show %USERPROFILE% and "…" instead of the profile folder and
user, computer or domain name. wofutil.dll is looked up at run time and
counts as not available under Wine, whose stubs end the process; the report
says so. The Wine scripts turn the crash dialog off, and the smoke test
checks that the console never shows the user name.

Skip read-only files, keep the record for unread files, never follow file links in the script ... 314a6228f4

The one-game test and measure-wof.ps1 now leave read-only files alone (skip reason read_only),
since restoring them may be refused. A restore counts a recorded file the walk did not list as
gone only where it no longer exists and the walk read every folder; otherwise the restore is
incomplete and the state file stays. measure-wof.ps1 no longer lists file symbolic links or
mount points, and its file facts never open one, so nothing outside the game folder is touched.

Merge branch 'worktree-wf_a4a9b4bb-de0-1' into windows-port ... 4533d70c60

# Conflicts:
#	README.md
#	docs/ARCHITECTURE.md

SkyfaR added 1 commit 2026-10-04 12:49:45 +02:00

Fix the probe's WOF detection after run 1, merge shared folders, correct LZX (probe 0.2.0) ... cbb6a05d4d

Run 1 on a Windows 11 PC showed WOF working (58033 files already LZX or
XPRESS4K) while the probe reported it missing on every drive, which
skipped the self-test, the calibration and the one-game test.

- FSCTL_GET_WOF_VERSION now gets its required WOF_EXTERNAL_INFO input and
  a WOF_VERSION_INFO output; WofGetDriverVersion is asked on its own. Both
  report the Win32 code where they fail (wof_version_error,
  wof_file_provider_error).
- The self-test starts with a trial (wof::trial): a 1 MiB file compressed
  through WofSetFileDataLocation and FSCTL_SET_EXTERNAL_BACKING, each with
  its code, the state read back, a content check and the undo, and the
  version queries before and after. The trial alone decides whether the
  test runs. It then calibrates the estimator on files of its own.
- Calibration and the one-game test only need NTFS and wofutil.dll; the
  real-WOF unit test is gated on the trial.
- Entries sharing one install folder are measured once under the first
  entry (shared_with); the others carry only same_folder_as. The one-game
  test uses the first entry's key.
- LZX model corrected by run 1's ground truth (saving x 1.13, at least
  72 % of the model). Raw model ratios stay in the report; files already
  WOF-compressed give ground_truth (real size against raw, corrected and
  final estimate), and the inventory lists compressed files per extension.
- Summary: WOF driver version or the query's code instead of "nicht
  aktiv", files already compressed per game and in total, shared folders,
  the trial's codes and the calibration factors.
- A report of another probe version is kept aside, not merged into.
- ANLEITUNG.txt for run 2, ARCHITECTURE.md, Wine smoke test with two app
  ids in one folder.

SkyfaR added 2 commits 2026-10-04 14:51:58 +02:00

Pass WOF the right struct, add the WOF diagnosis, guard the calibration (probe 0.3.0) ... 17f2b8c4a8

Run 2 and 2b compressed "XPRESS4K/8K" as LZX and failed LZX and XPRESS16K
with error 87. The cause: set_location passed the FSCTL's
FILE_PROVIDER_EXTERNAL_INFO_V1 {Version 1, Algorithm, Flags} (12 bytes) to
WofSetFileDataLocation, which takes WOF_FILE_COMPRESSION_INFO_V1
{Algorithm, Flags} (8 bytes). wofutil read Version 1 as the algorithm
(LZX) and the algorithm as the flags; 1 and 3 set flag bit 0
(compress-on-write), which WOF refuses with 87. The read-back parser is
right, so run 1's LZX labels and its x1.13 LZX correction stand.

- numbers::api_info / fsctl_info build both buffers; a Windows test
  compares them with windows-sys' structs.
- Every compress goes through wof::route::Router: API first, state read
  back, FSCTL where the API answers 87 or stores another algorithm.
  A wrong state is its own outcome (wrong_algorithm_<state>), never done;
  calls are counted per route and result.
- wof-diagnose (menu 8, and in the self-test after the trial): the
  matrix of the diagnosis brief. Blocks A (seven buffers, both routes,
  T and license.rtf), B (three handles), C (eight at once), D (switch),
  E (compact.exe control), each copy read back by the parser and by
  WofIsExternalFile, undone and deleted, and a verdict with the decision
  rules. Tested against a model of Windows' behaviour.
- Calibration: a factor only from a call that read back the algorithm
  asked for, only where the model saves at least 3 %, clamped to
  0.8-1.4; a calibrated LZX estimate stays above 72 % of the raw model.
- Throughput over compressed bytes only; H11/H12/H13 from real
  compressions; H6 is the median of the factors used.
- One-game test: files by outcome code, routes, kept "original" and the
  record removed where nothing changed; every launcher of a shared folder
  must be closed.
- WOF version shown as major.minor.build (10.0.10011).
- A report of another version is never written over (-2, -alt).
- Guide for run 3, ARCHITECTURE.md, Wine smoke test for wof-diagnose.

Undo a wrong algorithm before the other WOF route, add unverified (probe 0.3.0) ... 3e5418889b

The router read nothing before a compress call, so after the API stored the
wrong algorithm it tried the FSCTL on top; an FSCTL answer other than done
then left the file with the wrong algorithm and hid wrong_algorithm. Now it
reads the state first, removes a wrong algorithm with
FSCTL_DELETE_EXTERNAL_BACKING before the other route where the file was
plain, skips that route where it was not, and checks a final failure
against the file: stored otherwise than asked and than before, it is
wrong_algorithm with that state. Undos are counted as undo:<result>.

A successful call whose state cannot be read back is now unverified instead
of done, so it never passes the calibration guard or counts as compressed.

The diagnosis' A2 row passes 4 bytes from an 8-byte buffer, so a wofutil
that reads the whole struct stays in the probe's memory.

ANLEITUNG: steam:3441460 is LZX-compressed since run 2b although its record
says xpress8k; run 3 restores it with "zurueck" first (plain files come back
by removing the backing, never by compressing), then the new game tests.

SkyfaR referenced this pull request from a commit 2026-10-04 15:24:08 +02:00

Merge pull request 'Windows port, part 1: measurement probe (W0) and portability groundwork (W1)' (#29) from windows-port into main

SkyfaR manually merged commit f1c633f6d7 into main 2026-10-04 15:24:09 +02:00

SkyfaR referenced this pull request 2026-10-04 20:00:14 +02:00

Windows port, part 2: WOF engine, Windows platform and commands (W2) #31

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

LevelXStudios/OpenGameCompressor!29

No description provided.