Faster CI: prebuilt images, build jobs in parallel, cancel superseded runs #41

Manually merged

SkyfaR merged 4 commits from ci-speedup into main 2026-10-06 21:15:29 +02:00

Conversation 0 Commits 4 Files changed 14 +345 -94

SkyfaR commented 2026-10-06 19:13:26 +02:00

Owner

Copy link

Speeds up CI, as discussed:

1. Prebuilt CI images. Four images live in this org's container registry and are pinned in ci.yml by tag and digest: ogc-ci-arch, ogc-ci-alma8, ogc-ci-debian13 and ogc-ci-windows. Runs on pull requests, on main and on tags use them and skip the package installs. The recipes are in scripts/ci/; the Windows image is built from scripts/windows/Containerfile, so its package list stays in toolchain.sh.
   • Refresh: scripts/ci/build-images.sh is run locally, never by a workflow, so no job uses the runner's Docker socket. It builds --pull --no-cache, pushes, and prints name:tag@digest; --write puts the new references into ci.yml.
   • Freshness: a nightly scheduled run (03:47 UTC) uses the plain upstream images with live installs, so upstream breakage still shows within a day. vm.yml stays on upstream Arch.
2. The build jobs no longer wait for check: package, bundle-cli, bundle-gui and the Windows jobs start at once. bundle waits for its two program jobs. The release (tags only) still waits for all 13 jobs.
3. Superseded runs are cancelled: a newer push to the same branch or PR cancels the older run. Tag runs and the nightly run are never cancelled.

Local check: each job ran in its image on a fresh clone with docker run: check (with the GUI tests on Broadway), package, bundle-cli, bundle-gui, bundle, windows-check, windows-sysroot and windows-gui-check. All passed.

Measurement: before, run 145 took 10:41. Its critical path was check (5:47), then bundle-gui (3:11), then bundle. The run of this PR gives the "after" figure. The first run on a runner host also pulls about 3.4 GB of images once.

Not verified yet on the runner: the expression in container.image that picks the nightly upstream image, and concurrency. Both match the Forgejo docs; this PR's run and the next nightly run will show them.

Speeds up CI, as discussed: 1. **Prebuilt CI images.** Four images live in this org's container registry and are pinned in `ci.yml` by tag and digest: `ogc-ci-arch`, `ogc-ci-alma8`, `ogc-ci-debian13` and `ogc-ci-windows`. Runs on pull requests, on `main` and on tags use them and skip the package installs. The recipes are in `scripts/ci/`; the Windows image is built from `scripts/windows/Containerfile`, so its package list stays in `toolchain.sh`. - **Refresh:** `scripts/ci/build-images.sh` is run locally, never by a workflow, so no job uses the runner's Docker socket. It builds `--pull --no-cache`, pushes, and prints `name:tag@digest`; `--write` puts the new references into `ci.yml`. - **Freshness:** a nightly scheduled run (03:47 UTC) uses the plain upstream images with live installs, so upstream breakage still shows within a day. `vm.yml` stays on upstream Arch. 2. **The build jobs no longer wait for `check`:** `package`, `bundle-cli`, `bundle-gui` and the Windows jobs start at once. `bundle` waits for its two program jobs. The release (tags only) still waits for all 13 jobs. 3. **Superseded runs are cancelled:** a newer push to the same branch or PR cancels the older run. Tag runs and the nightly run are never cancelled. **Local check:** each job ran in its image on a fresh clone with `docker run`: `check` (with the GUI tests on Broadway), `package`, `bundle-cli`, `bundle-gui`, `bundle`, `windows-check`, `windows-sysroot` and `windows-gui-check`. All passed. **Measurement:** before, run 145 took 10:41. Its critical path was `check` (5:47), then `bundle-gui` (3:11), then `bundle`. The run of this PR gives the "after" figure. The first run on a runner host also pulls about 3.4 GB of images once. **Not verified yet on the runner:** the expression in `container.image` that picks the nightly upstream image, and `concurrency`. Both match the Forgejo docs; this PR's run and the next nightly run will show them.

SkyfaR added 3 commits 2026-10-06 19:13:26 +02:00

Add recipes and a build script for prebuilt CI images ... 9537c6ad13

scripts/ci/ holds one Containerfile and install script for each CI
image (Arch, AlmaLinux 8, Debian 13); the Windows image is
scripts/windows/Containerfile, the one in-container.sh uses, so its
package list stays in toolchain.sh. build-images.sh builds them from the
newest upstream images, pushes them to Forgejo's registry and prints the
name:tag@digest references for ci.yml (--write puts them there).

Run CI in prebuilt images, without waiting for check, cancelling old runs ... 738319dcf8

Pull request, main and tag runs use the prebuilt images pinned by tag and
digest, so no job installs packages any more. A nightly scheduled run
keeps the plain upstream images and installs everything live with the
images' own scripts, so changes upstream still show within a day.

package, bundle-cli and bundle-gui no longer wait for check; release
waits for every job. A newer push to a branch or pull request cancels
the run still going for the one before, but tag runs are never
cancelled. vm.yml cancels superseded runs too.

docs/ARCHITECTURE.md describes the images, how to refresh them, the
nightly run, the concurrency and the job graph.

Rebuild CI images without Docker's layer cache, and log out after pushing ... 2b3a6a7b5b

A refresh reused the cached install layer whenever the upstream image had
not changed, so it pushed the old packages under a new tag. The script
now builds without the cache and says when a digest equals the pinned
one; the docs log Docker out of the registry after pushing.

SkyfaR added 1 commit 2026-10-06 20:25:12 +02:00

Wait in the tests until a started sleep runs, also for its environment ... fe0877d606

A child reads as the test program, with its name and environment, until it
has exec'd sleep. tells_many_folders_their_users_at_once failed in CI when it
read the Proton child's environment before that.

SkyfaR manually merged commit aa130d41ec into main 2026-10-06 21:15:29 +02:00

SkyfaR referenced this pull request from a commit 2026-10-06 21:15:30 +02:00

Merge pull request 'Faster CI: prebuilt images, build jobs in parallel, cancel superseded runs' (#41) from ci-speedup into main

SkyfaR referenced this pull request 2026-10-06 21:31:35 +02:00

Faster CI, part 2: Windows build without waiting, VM test only for code the guest builds #42

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

LevelXStudios/OpenGameCompressor!41

No description provided.