ext4 container, P0: test package for SteamOS, VM scenarios, SteamOS VM #46

Manually merged

SkyfaR merged 33 commits from ext4-p0 into main 2026-10-07 07:31:32 +02:00

Conversation 0 Commits 33 Files changed 56 +7819 -26

SkyfaR commented 2026-10-07 03:41:41 +02:00

Owner

Copy link

Measurement tooling for the ext4 container plan (P0). Nothing in the app changes.

• ogc-decktest (scripts/decktest/, tools/ogc-deckprobe/, built by scripts/decktest/build.sh): a test package for a SteamOS device covering E1–E5, E8, E9, E12–E14. German, impersonal guide. Every root step asks first and uses one sudo; everything lives under /home/.ogc-decktest plus one test unit; aufraeumen removes it all. It never deletes a test library that holds a game, mounted or not (found in review), grows the library when the next game does not fit, and the report strips user/host names, machine id, MACs, UUIDs and the home path.
• VM scenarios (tests/vm/, scripts/vm-e2e.sh <name>): container-discard (E5), container-enospc (E6), container-crash (E7, dm-log-writes), container-powercut (E7), container-churn (E11), decktest (the package end to end). The existing seven still pass.
• SteamOS VM (scripts/steamos-vm.sh): Valve's 3.8.14 recovery image pinned by checksum, installed without root; scenarios for the package, factory reset, reinstall and an OS update.

First results (write-ups in the private plans): E6 fails model B (19/25 runs went read-only) and passes model A (clean ENOSPC), which confirms model A; E7 crash replay 300/300 and hard kills 20/20; E9 update 3.8.14 → 3.8.28 keeps unit and link; a factory reset removes everything; a reinstall keeps the image but loses the unit.

Review: 9 findings (1 high) and 6 from the re-review, all fixed and re-checked.

Measurement tooling for the ext4 container plan (P0). Nothing in the app changes. - **`ogc-decktest`** (`scripts/decktest/`, `tools/ogc-deckprobe/`, built by `scripts/decktest/build.sh`): a test package for a SteamOS device covering E1–E5, E8, E9, E12–E14. German, impersonal guide. Every root step asks first and uses one `sudo`; everything lives under `/home/.ogc-decktest` plus one test unit; `aufraeumen` removes it all. It never deletes a test library that holds a game, mounted or not (found in review), grows the library when the next game does not fit, and the report strips user/host names, machine id, MACs, UUIDs and the home path. - **VM scenarios** (`tests/vm/`, `scripts/vm-e2e.sh <name>`): container-discard (E5), container-enospc (E6), container-crash (E7, dm-log-writes), container-powercut (E7), container-churn (E11), decktest (the package end to end). The existing seven still pass. - **SteamOS VM** (`scripts/steamos-vm.sh`): Valve's 3.8.14 recovery image pinned by checksum, installed without root; scenarios for the package, factory reset, reinstall and an OS update. First results (write-ups in the private plans): E6 fails model B (19/25 runs went read-only) and passes model A (clean ENOSPC), which confirms model A; E7 crash replay 300/300 and hard kills 20/20; E9 update 3.8.14 → 3.8.28 keeps unit and link; a factory reset removes everything; a reinstall keeps the image but loses the unit. Review: 9 findings (1 high) and 6 from the re-review, all fixed and re-checked.

SkyfaR added 33 commits 2026-10-07 03:41:41 +02:00

Add ogc-deckprobe, the measurement probe of the SteamOS test package ... 856d3c9301

A crate of its own (as ogc-probe is): Steam's games, plain copies, E1's read
patterns with CPU, disk and page cache counters, the zstd levels in user
space, sample and game-like data, a process tree's reads for the launch time,
memory pressure, and the report without identifiers.

Add the SteamOS test package ogc-decktest ... 4a86c3ec77

decktest.sh (German, impersonal) runs E1 to E5, E8, E9, E12 and E13 of the
ext4 container plan on the friend's device in Desktop Mode: every step that
needs root asks first and then runs lib/root.sh through sudo, everything it
creates lives below /home/.ogc-decktest and in one test unit, and aufraeumen
removes all of it. build.sh builds the static probe in a container and packs
ogc-decktest-<version>.tar.gz with its .sha256; tests.sh tests the driver
without root against a made-up home.

Add ogc-logreplay, a replayer of dm-log-writes logs for the VM test ... cc4a52a6c0

Lists a log's entries with their offsets, flags and marks, and replays a
range of them onto a device, discards as zeros, so that the crash
measurement can check the state at every flush point.

Measure the ext4 container and run the SteamOS test package in the VM test ... 6fa82f668f

New scenarios, run only when named: container-discard (E5),
container-enospc (E6), container-crash (E7 with dm-log-writes),
container-powercut (E7 with qemu killed), container-churn (E11) and
decktest (the package end to end, across a reboot, with a fake Steam).
boot.sh gives them a larger data disk with cache=none, a longer unit time
through ogc.timeout= and a generator, and kills qemu when a guest asks for
it. The existing scenarios boot as before. OGC_E2E_IMAGE names the
container image, so that two work trees do not build over each other.

Add a SteamOS VM from Valve's recovery image ... 993ec8310f

scripts/steamos-vm.sh fetches the pinned recovery image, installs SteamOS
onto an empty NVMe disk with the recovery's own repair_device.sh (headless
through NOPROMPT, the Deck's BIOS and controller updates replaced through
its vendored folders), and runs scenarios with a driver that a test unit in
the /etc overlay starts at boot: the test package end to end, a factory
reset and "Reinstall SteamOS" with the test library set up. Files go into
the ext4 partitions with debugfs, so nothing needs root.

ogc-deckprobe: judge only C1 to C6 by E1's rule, and E3 under pressure ... 6a7c5c32d2

The game where it is (C0') was judged as if it were a configuration of the
container. The report now also compares buffered with direct I/O under
memory pressure, as E3 asks.

decktest: stamp the package with the time of its own last change ... f1cd191997

So that commits elsewhere do not change the tarball's bytes.

VM test: record the kernel's messages of E6 and whether the loop's discard switch can be undone ... 477421db3d

container-enospc-kurz makes one run of each kind with the kinds of kernel
messages (a loop device that meets ENOSPC reports a space allocation error,
not an I/O error), the writer's error keeps its reason, and the full E6 gets
the time its 41 runs take. E5 measures the preallocated image with the switch
left on first, as a loop device keeps a switched-off discard after its
detach, and tries to switch it back.

Describe the SteamOS test package, the container measurements and the SteamOS VM ... f9cdf57213

In the architecture notes: ogc-deckprobe and decktest.sh, the VM scenarios
container-* and decktest with what boot.sh gives them, and how the SteamOS
VM installs and drives SteamOS from Valve's recovery image.

Remove loop devices after use, as their switched-off discard outlives a detach ... df9ab3abf0

E5 showed that a loop device keeps discard_max_bytes = 0 after losetup -d
and that writing the device's own limit back has no effect. ogc-deckprobe
gets loop-entfernen (LOOP_CTL_REMOVE); the test package removes every loop
device it detached, also the test unit at shutdown, and E5 checks that a
device made anew starts with discard again.

ogc-deckprobe: keep a label as text unless it reads back as the same number ... 187a952f64

A SteamOS build id such as 20260707.10 became the number 20260707.1 in the
records of dauerhaft.

SteamOS VM: an update scenario for E9 ... b6efebb77e

The test library and its unit set up, then a SteamOS update (stable, else
once beta), a reboot into the new build and the package's check. The driver
writes its sudoers entry again at every boot, as the keep list does not keep
/etc/sudoers.d.

Describe the SteamOS VM's update scenario ef244bd820

VM test: let loop_queue report a refused sysfs write ... beae976ace

loop_queue returned cat's status, so that the 'refused' fallbacks of its
callers could never take effect: E5 recorded the switch-back of the loop's
discard as accepted although the kernel refused it. The write's failure is
now returned, and the switch-back records the error and the device's
discard_granularity, which decides whether a non-zero value is taken.

VM test: time a cap shrink that has to move chunks (E6 step 3, R18) ... ee201a0091

The cap shrink resized to the cap untimed first, which is where any
relocation happened, and then timed small shrinks with nothing above them;
these counted as passing toward 'every cap shrink took 30 s or less'.

Each case now takes a fresh copy of the churned image, fills it to 70 % of
the guard's cap and times one shrink straight from the image's full size
to what btrfs uses plus 1 GiB, once with room on ext4 and once with ext4 at
the reserve + 1 GiB. The end of the last device extent is recorded before
and after; a shrink with nothing above its target counts as relocated=no
and is left out of the verdict, which says so. container-shrink runs the
churn and the shrinks alone.

ogc-deckprobe: show E1's spread across the runs and mark a noisy verdict ... 9263a0fc31

bericht.txt gave each group's median only, so a verdict such as 1.08
against 1.10 could not show whether it lay within the run-to-run noise.
The table now has the smallest and largest run and the coefficient of
variation per group, and an E1 verdict says 'unsicher' when the fastest
run of the configuration against the slowest of C0 is within the limit
and the slowest against the fastest is not.

ogc-deckprobe: scale E2's end of loading by the game's on-disk factor ... b4e93c0498

read_bytes counts what the game's reads fetch from the filesystem, which on
a compressed btrfs are the compressed extents; the loop device's reads of
the image are a kernel worker's. The fixed thresholds (50 MB, then below
1 MB/s for 5 s) therefore fired at a higher real read rate in the container
than on ext4, and a small game might never reach the minimum there.

folgen takes --faktor (disk bytes ÷ logical bytes of the moved game, 1 on
ext4) and scales both thresholds by it; it records the factor, the
thresholds used and rchar beside read_bytes, so that the evaluation can
check the scaling. The new command belegung measures the factor: it reads
every file's extent items with BTRFS_IOC_TREE_SEARCH, as compsize does
(root only), and also gives the share of compressed extents.

decktest: label E2's launches by what komprimieren did, and scale by the game's factor ... e5a889a027

ladezeit ran komprimieren with '|| true' and labelled the next launches
C<level> whatever happened: a declined sudo or a failed defragment still
gave four 'compressed' launches of a game that lay in the library as
Steam's Move wrote it, and C<level> did not match the plan's names (zstd:1
is C2, not C1, and direct I/O is C4/C5).

komprimieren now records what the game takes on disk before and after
(ogc-deckprobe belegung: disk ÷ logical bytes, compressed extents), and
ladezeit reads that record back: the launches are named as the plan names
the configuration (C2 to C6), as zstd-standard when btrfs took no level,
or with -nur-verschoben when the game was not compressed. The game's
factor goes to folgen, so that E2's end of loading fires at the same point
of the real loading as on ext4. The label logic is tested with a stand-in
for the root step.

decktest: grow the test library when the next game does not fit ... daa931d36e

The library was sized only when ladezeit or dauerhaft created it, so a
second, larger game (the plan's G2 has 20-50 GB) did not fit: Steam's Move
failed with only Steam's message, after the four launches from ext4, and
the step polled for two hours before it gave up.

Before the first launch ladezeit now compares the library's free space
with the game's size (+5 %) and has the new root verb
bibliothek-vergroessern grow it by what is missing and 1 GiB (fallocate or
truncate, losetup -c, btrfs resize max), when /home keeps 4 GiB besides;
otherwise it stops with the sizes. After five minutes of waiting for a
Move, the step says how to stop it if Steam shows an error.

decktest: release a failed or stopped measurement's images and mounts ... bd80247fd1

root.sh trapped EXIT only to give the results back. ENOSPC in a copy, a
btrfs error or Ctrl+C in lesen, zstd or trim left a test image of up to
1.25 × the game attached and mounted below work/, holding /home's space; a
rerun deleted the attached image's name and mounted over the old mount.

For these verbs the EXIT trap now stops the memory balloon, unmounts
everything below work/ (deepest first), detaches and removes every loop
device whose image lies there and deletes the images and copies; HUP, INT
and TERM end the step through it. The test library is outside work/ and
stays. A failed root step's message names aufraeumen for space still held.

VM test: check komprimieren's record, E2's factor and a stopped lesen ... a1f4b897d1

The decktest scenario now checks that the moved game was compressed and
measured on disk with a factor below 1, that its four launches are named
C3 and carry that factor while C0's carry 1, and that rchar is recorded.
It also stops lesen's root step with SIGTERM while a test image is mounted
and expects no mount, loop device or image below work/ afterwards, the
hint to aufraeumen, and the test library still mounted.

decktest: never delete a test library that holds a game, mounted or not ... b5758b402a

aufraeumen looked for appmanifest files only while the library was
mounted. After a reboot following ladezeit (no unit), a failed mount, the
late-mount test or an unmount through udisks, Steam lists the library as
missing, the game in it cannot be moved back, and aufraeumen, answered
yes, deleted the image with the friend's real game and possibly its
Proton prefix inside. The guide even said it may run at any time.

The root step now looks into an unmounted library read-only (its own
read-only loop device, rescue=nologreplay, below work/) and refuses while
it holds an appmanifest, a game folder or a Proton prefix (compatdata,
save games), and also when the image cannot be mounted to look. The
user's side checks a mounted library first. The refusal says what to do:
the new command einhaengen mounts the library again so that Steam can
move the game back; a library with only a prefix left goes with
aufraeumen --abbild-loeschen, which asks once more. The guide says so.

The VM test puts a game into the library, stops the unit, expects
aufraeumen to refuse and keep the image with no mount or loop device
left, mounts the library with einhaengen, moves the game out, expects a
refusal for the prefix, and then cleans up with --abbild-loeschen.

SteamOS VM: let aufraeumen look into an unmounted library in the package run ... 8738306649

The paket scenario stops the test unit before aufraeumen, so that the
read-only look into an unmounted library runs on SteamOS's own kernel and
tools, and checks that it was announced and that nothing is left.

decktest: refuse rather than skip a pending btrfs log when looking into the library ... 00f118c091

The read-only look of aufraeumen mounted with rescue=nologreplay, which
would hide files fsynced just before a crash (a manifest perhaps). It now
mounts read-only without that option: on the read-only loop device btrfs
refuses a filesystem whose log needs replaying, and aufraeumen refuses
with it and names einhaengen, after which it checks the mounted library.

VM test: expect the stopped lesen's failure in its output, not its status ... c497527462

decktest.sh goes on to its report after a failed root step, so the
command ends with 0; the test now looks for the failure message.

VM test: record why a cap shrink failed and how many block groups it moved ... 18ac79cab9

The first run's failed shrink recorded btrfs's progress line instead of
its error; the error line and the count of relocated block groups from
the kernel's messages are now recorded, and the resize's output is
printed.

decktest: name aufraeumen for held space only after a failed measurement ... 6bbe7c8aee

A refused aufraeumen or another failed root step said that aufraeumen
frees space still held, which only fits lesen, zstd and trim.

Describe the test package's fixes from the review in ARCHITECTURE.md ... 92cd6f4876

belegung and the scaled end of loading, the spread in the E1 summary,
einhaengen, the guarded aufraeumen with --abbild-loeschen, the clean-up
of a stopped measurement, the growing library and the launch names.

VM test: throttle the E6 filler ... f3d88a55b6

awk read the `>` of `printf "%.3f", d > 0 ? d : 0` as an output
redirection and failed, so the wait came out empty and the filler wrote
as fast as the VM could in every run. Parenthesised.

decktest: look into an unmounted library before Steam is asked to drop it ... b278601e32

aufraeumen asked to remove the test library in Steam before root had
looked into it; when the unmounted library still held a game, the
refusal came after the user had removed it, and nothing listed it in
Steam again, so the game could not be moved back. aufraeumen now asks
for that only while the library is mounted and otherwise says not to
remove it yet; einhaengen has a running Steam list the library again
(steam://addlibraryfolder) and says what to do when it does not.
Tested in tests.sh and in the VM's aufraeumen_guards.

decktest: measure a declined compression's game, and say when E2 is unscaled ... 5dcd4ebc23

When komprimieren was declined, ladezeit scaled E2's thresholds by 1,
although Steam's Move under compress=zstd had compressed part of the
game. A small root step (belegung) now measures the game as the Move
left it, changing nothing. faktor_gemessen follows whether a factor was
measured, not whether komprimieren wrote a record, so a failed belegung
inside komprimieren is no longer recorded as measured.

decktest: refuse ladezeit settings other than the existing library's ... f0f76334b6

An existing test library (from an earlier ladezeit or from dauerhaft
einrichten) is mounted with the settings it was made with, so
`ladezeit --direkt` against a buffered one measured C3 and labelled it
C5. ladezeit now refuses a level or direct I/O other than the library's
before any launch or sudo and names the command that matches it, or the
clean-up that lets it make a new one.

decktest: grow the library's own loop device, tested in the VM ... 4c5d335709

bibliothek-vergroessern took the loop device from findmnt's SOURCE,
which may carry a [subvolume] suffix; it now asks losetup for the
image's device. The VM scenario runs the real root step and checks the
image (preallocated), the loop device, btrfs's device size, the room
in the library and bibliothek.txt.

SkyfaR referenced this pull request from a commit 2026-10-07 07:31:32 +02:00

Merge pull request 'ext4 container, P0: test package for SteamOS, VM scenarios, SteamOS VM' (#46) from ext4-p0 into main

SkyfaR manually merged commit 19ad7868e9 into main 2026-10-07 07:31:32 +02:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

LevelXStudios/OpenGameCompressor!46

No description provided.