#+TITLE: M1: Multi-Kernel Test Framework — Step Plan
#+DATE: 2026-03-08 (completed 2026-03-15)

*Status: DONE.* All phases complete. Branch =m1-testing= pushed.
Merges to =master= as part of v1.7.0 release (after M3 bug fixes).

* Overview

Build a local-first, CI-agnostic test framework that tests quotatool
across 15-20+ Linux kernel versions (including vendor-patched kernels)
and two filesystem types (ext4, XFS).

Architecture:
#+begin_example
test/                          ← workspace (all test framework code)
  run-tests                 ← entry point: "run one command, get results"
  lib/
    boot.sh                    ← kernel boot layer (virtme-ng / raw QEMU)
    fs-setup.sh                ← loopback filesystem create/mount/teardown
    tap.sh                     ← TAP output helpers
  tests/
    t-block-limits.sh          ← individual test scripts
    t-inode-limits.sh
    t-grace-period.sh
    ...
  kernels/
    kernels.conf               ← matrix definition (name, vmlinuz path, boot method)
    download.sh                ← fetch + extract vendor kernels
    initramfs/                 ← custom initramfs for legacy kernels
  results/                     ← output (gitignored)

.github/workflows/test.yml    ← thin CI wrapper (calls run-tests)
.cirrus.yml                   ← FreeBSD testing
#+end_example

Background: =research/testing-infrastructure.org=,
            =research/multi-kernel-testing.org=,
            =research/quotatool-build-analysis.org=

Key finding: quotatool ships its own quota struct definitions
(=src/linux/linux_quota.h=, =xfs_quota.h=, =dqblk_old.h=, =dqblk_v0.h=)
and does runtime detection of the kernel's quota interface in
=src/linux/quota.c:kern_quota_format()=. *One binary works across all
kernel versions.* Build once, test everywhere. But the runtime detection
code is the critical fragile point — it was last updated around the 4.x
kernel era and likely needs fixes for modern kernels.

* Progress tracking

Each task has an org-mode TODO keyword:
=TODO=, =DOING=, =WAITING=, =DONE=, =CANCELLED=.
Update status in this file at every checkpoint. This file is the
single source of truth for M1 progress.

* Phase 1: Foundation [3/5]

Minimal working loop: boot ONE kernel, run ONE test, get a result.
Everything else builds on this.

** DONE 1.1 Create workspace and directory structure
Set up =test/= directory tree as shown above. Skeleton files only.
/Verification/: =ls -R test/= shows expected layout.

** DONE 1.2 Build the kernel boot layer (=test/lib/boot.sh=)
A function that takes a kernel path and a test script, boots QEMU,
runs the script inside the VM, returns exit code + stdout.

Two code paths:
- *Modern (≥5.4)*: virtme-ng with virtiofs
- *Legacy (<5.4)*: raw QEMU with =-kernel=, custom initramfs,
  9p filesystem sharing (msize=524288)
- *RHEL (no 9p)*: raw QEMU with virtio-blk rootfs disk image

/Subtasks/:
- [ ] Detect virtme-ng availability, fall back to raw QEMU
- [ ] Modern path: =vng -r <vmlinuz> --disk <test-disk> -- <script>=
- [ ] Legacy path: QEMU with 9p share + =-kernel= + initramfs
- [ ] Exit code propagation from guest to host
- [ ] Timeout handling (kill hung VMs)
- [ ] Verify on host kernel first (=vng -- <script>=)

/Verification/: =boot.sh /boot/vmlinuz-$(uname -r) "uname -r"=
prints the kernel version and exits 0.

/Dependencies/: 1.1

** DONE 1.3 Build filesystem setup/teardown (=test/lib/fs-setup.sh=)
Functions to create loopback ext4/XFS images with quotas enabled.
Runs *inside* the VM.

/Subtasks/:
- [ ] =fs_create_ext4 <path> <size>= → loopback + mkfs + mount + quotaon
- [ ] =fs_create_xfs <path> <size>= → loopback + mkfs + mount (no quotacheck)
- [ ] =fs_teardown <path>= → quotaoff + umount + cleanup
- [ ] Error handling: clean teardown on failure

/Verification/: Inside a VM, =fs_create_ext4 /mnt/test 100M= creates
a working quota-enabled filesystem. =repquota /mnt/test= succeeds.

/Dependencies/: none (runs inside VM, no boot layer needed to write)

** DONE 1.4 Dependency check script (=test/check-deps.sh=)
A script that verifies all required tools are present and working.
Serves two purposes: (1) verify our dev environment before testing,
(2) help anyone cloning the repo to set up for running tests.

Replaces the static =test/DEPENDENCIES.md= with an executable check.

/Subtasks/:
- [ ] Check for: qemu-system-x86_64, virtme-ng/vng, /dev/kvm access
- [ ] Check for: mkfs.ext4, mkfs.xfs, quota tools (quotaon, repquota)
- [ ] Check for: losetup, mount (with loop support)
- [ ] Print clear pass/fail per dependency with install hints
- [ ] Exit 0 only if all required deps are present

/Verification/: =./test/check-deps.sh= passes on a correctly set up host,
fails with helpful messages on a bare system.

/Dependencies/: 1.1

** DONE 1.5 First end-to-end test
Combine 1.2 + 1.3: boot host kernel, create ext4 loopback inside VM,
run =quotatool -u nobody -b -l 100M /mnt/test=, verify with =repquota=.

/Verification/: =./test/run-tests= boots VM, runs one test, reports
PASS or FAIL, exits with correct code.

/Dependencies/: 1.2, 1.3, 1.4

* Phase 2: Test Suite [0/5]

Write the actual test cases. Each test is a self-contained function
that assumes a quota-enabled filesystem is mounted.

** DONE 2.1 Define test case list
Based on quotatool's features. Reference =research/PLAN.org= test
case table. Each test must be independently runnable.

Minimum set:
| ID   | Test                            | FS        | Category |
|------+---------------------------------+-----------+----------|
| B01  | Set user block soft limit       | ext4, XFS | basic    |
| B02  | Set user block hard limit       | ext4, XFS | basic    |
| B03  | Set group block soft/hard limit | ext4, XFS | basic    |
| I01  | Set user inode soft limit       | ext4, XFS | basic    |
| I02  | Set user inode hard limit       | ext4, XFS | basic    |
| U01  | Human-readable units (K,M,G,T)  | ext4      | parse    |
| U02  | Relative adjustment (+100M)     | ext4      | parse    |
| F01  | Raise-only flag (-R)            | ext4      | feature  |
| F02  | Grace period set (-t)           | ext4      | feature  |
| F03  | Grace period restart (-r)       | ext4      | feature  |
| F04  | Numeric UID (:12345)            | ext4      | feature  |
| O01  | Verbose output (-v)             | ext4      | output   |
| O02  | Dump/machine-readable (-d)      | ext4      | output   |
| E01  | No root → graceful failure      | ext4      | error    |
| E02  | Invalid arguments               | ext4      | error    |
| E03  | Non-existent mount point        | ext4      | error    |
| D01  | Format detection succeeds       | ext4      | detect   |
| D02  | Format detection succeeds       | XFS       | detect   |
| D03  | Get+set round-trip (set, get back, compare) | ext4, XFS | detect |

*Category "detect"*: These test =kern_quota_format()= — the runtime
format detection in =src/linux/quota.c=. This is the most fragile code
path and the primary reason for multi-kernel testing. It probes
=/proc/fs/quota=, =/proc/sys/fs/quota=, and falls back through
=Q_V0_GETSTATS= → =Q_OLD_GETSTATS= → =Q_OLD_GETQUOTA=. It contains
vendor-specific workarounds (e.g., RedHat 2.4.2-2) and has not been
updated since ~kernel 4.x. *Any kernel where detection fails = test
failure = investigation needed.* These tests are the early warning
system for the entire project.

/Verification/: test list reviewed, each test has clear pass criteria.

** CANCELLED 2.2 Write TAP output layer (=test/lib/tap.sh=)
TAP (Test Anything Protocol) for machine-parseable output.
Simple: =ok 1 - description= / =not ok 2 - description=.

/Verification/: output parseable by standard TAP consumers.

** DONE 2.3 Write test implementations
Implement each test from 2.1 as a shell function. Each test:
1. Calls quotatool with specific args
2. Checks exit code
3. Verifies result via =repquota= or =quota=
4. Emits TAP line

/Parallelizable/: tests can be written by multiple agents, each
handling a category (basic, parse, feature, output, error).

/Dependencies/: 2.1, 2.2

** DONE 2.4 Filesystem parameterization
Each test runs against both ext4 and XFS (where applicable).
The test runner creates both filesystem types and loops tests.

/Dependencies/: 2.3, 1.3

** DONE 2.5 Validate test suite on host kernel
Run full suite inside a single VM with the host's own kernel.
All tests must pass before moving to multi-kernel.

/Verification/: =./test/run-tests= reports all tests passing.

/Dependencies/: 2.3, 2.4, 1.5

* Phase 2b: Host-Side Tests [0/4]

Tests that do NOT need the kernel matrix. Two categories:

- *Quick tests* (no root, no VM, no filesystem): argument validation,
  error handling, exit codes. Run directly on host, instant.
- *Value tests* (need quota filesystem = need VM): unit parsing,
  round-trip set-then-readback, dump format. Run via =--host-only=
  (boots host kernel in VM, ~5s).

Both run *before* the kernel matrix in a normal =run-tests= run.

Research basis: source analysis of =parse.c= and =main.c=, cross-
referenced with GitHub issues #1, #2, #6, #7, #8, #10, #27. Every
test below exercises a code path with documented bug history or
verified surprising behavior.

** DONE 2.6 Quick tests: argument validation (no root, no VM)

All of these fail in =parse_commandline()= or early =main()= —
before any =quotactl()= call. Testable directly on host without
root. Verified: all produce exit code 1 (=ERR_PARSE=) or 2
(=ERR_ARG=) with correct error messages on stderr.

Location: =test/tests/host/t-error-args.sh=

Tests (each checks exit code + error message on stderr):

| Test                              | Expected exit | Code path              |
|-----------------------------------+---------------+------------------------|
| No arguments                      |             1 | =argc==1= → help       |
| =-q= without =-b/-i=             |             1 | =_PARSE_UNDEF= check   |
| =-l= without =-b/-i=             |             1 | same                   |
| =-t= without =-b/-i=             |             1 | same                   |
| =-r= without =-b/-i=             |             1 | same                   |
| Both =-u= and =-g=               |             1 | =id_type= already set  |
| No filesystem argument            |             1 | =qfile= NULL check     |
| No =-u/-g= at all                |             1 | =id_type= 0 check      |
| =-t= mixed with =-l=             |             1 | grace+limit conflict   |
| =-r= mixed with =-l=             |             1 | reset+limit conflict   |
| Unknown option (=-Z=)            |             1 | =getopt= =?= case      |
| Nonexistent user (no =:= prefix) |             2 | =system_getuid()= fail |
| =:= with non-numeric (=:abc=)    |             2 | =strtol= → id 0?      |
| =:= alone (empty after colon)    |           1/2 | edge: =strlen>1= false |

/Verification/: script exits 0 if all sub-tests pass. Each test
is a simple =quotatool <bad-args>; check $? and stderr=. No root.

** DONE 2.7 Value tests: unit parsing (VM, host kernel)

These need a quota filesystem for set-then-readback. Run inside VM
via =--host-only= path. All use =quotatool -d= to read back values
and compare against expected.

Location: =test/tests/host/t-parse-units.sh= (blocks + inodes)
         =test/tests/host/t-parse-timespan.sh= (grace periods)
         =test/tests/host/t-dump-format.sh= (=-d= output stability)

*** Block unit parsing (=parse_size= with =PARSE_BLOCKS=)

Each test: set limit, read back with =-d=, compare field value.

| Input       | Expected blocks | Why this matters                         |
|-------------+-----------------+------------------------------------------|
| =100M=      |          102400 | Standard, baseline (already tested)      |
| =1K=        |               1 | Minimum: 1024 bytes / 1024 BLOCK_SIZE    |
| =1G=        |         1048576 | Issue #1: Gb was missing pre-1.4.12      |
| =1T=        |      1073741824 | Issue #8: overflow at 4T+                |
| =1.5M=      |            1536 | Float: exercises =strtod= path           |
| =1.1T=      |      1181116006 | Issue #8: =strtol= missed decimal        |
| =0.5G=      |          524288 | Sub-unit fraction                        |
| =1024bytes= |               1 | =by= suffix → multiplier 1, ÷1024=1     |
| =1blocks=   |               1 | =bl= suffix → multiplier BLOCK_SIZE      |
| =100=       |             100 | No suffix → default = blocks             |

*** Inode unit parsing (=parse_size= with =PARSE_INODES=)

Critical: inodes use *base-10* multipliers, not base-2 (issue #10).

| Input  | Expected inodes | Why this matters                     |
|--------+-----------------+--------------------------------------|
| =1K=   |            1000 | Must be 1000, NOT 1024               |
| =1M=   |         1000000 | Must be 10^6, NOT 2^20               |
| =1G=   |      1000000000 | Must be 10^9, NOT 2^30               |
| =1.5K= |            1500 | Float × base-10                      |
| =100=  |             100 | No suffix → default = 1 (raw inodes) |

*** Relative adjustment edge cases

| Scenario                         | Expected                 | Bug potential             |
|----------------------------------+--------------------------+---------------------------|
| Set 100M, then =+0M=            | Stays 102400 (no-op)     | *Likely BUG*: returns 0   |
| Set 100M, then =-0M=            | Stays 102400 (no-op)     | Same bug path             |
| Set 100M, then =+100M=          | 204800                   | Sanity check              |

The =+0= bug: =parse_size()= line 475 returns 0 when =count==0=,
ignoring =orig= and =op=. So =+0M= applied to 100M → 0, not 100M.

*** Timespan parsing (=parse_timespan=)

Set grace with =-t=, read back with =-d=, compare seconds.

| Input         | Expected seconds | Why                         |
|---------------+------------------+-----------------------------|
| =3600 seconds= |           3600 | Explicit seconds            |
| =60 minutes=  |            3600 | =mi= prefix                |
| =1 hour=      |            3600 | Same value, different unit  |
| =1 day=       |           86400 | =DAY= macro                |
| =2 weeks=     |         1209600 | Issue #27: failed pre-1.6.4 |
| =1 month=     |         2592000 | =mo= prefix                |
| =5m=          |          *error* | Ambiguous: not mi, not mo   |
| =5min=        |             300 | Unambiguous abbreviation   |
| =5mo=         |        12960000 | Unambiguous month           |

*** =-d= dump format stability

| Check                         | Expected | Why                     |
|-------------------------------+----------+-------------------------|
| Field count                   | 10       | Issue #7: format bug    |
| All fields numeric            | yes      | Regression guard        |
| Fields not concatenated       | yes      | Issue #7: missing space |
| =-R= with equal value         | no change | =<== vs =<= semantics |

** DONE 2.8 Integration: wire into run-tests

- New =--quick= flag: runs only =test/tests/host/t-error-args.sh=,
  no VM, no root, instant. For rapid dev feedback.
- Normal run: execute quick tests first, then =--host-only= value
  tests, then kernel matrix (if not =--quick= or =--host-only=).
- Exit early if quick tests fail (parse bugs → no point running VMs).

/Dependencies/: 2.6, 2.7

** DONE 2.9 Verify tests catch regressions

For each test category, deliberately break the source code, rebuild,
and confirm the test catches the breakage:

- Swap inode multiplier to base-2 → =t-parse-units.sh= must fail
- Remove =-b/-i= validation → =t-error-args.sh= must fail
- Break =-d= field format → =t-dump-format.sh= must fail
- Break timespan =mi= match → =t-parse-timespan.sh= must fail

This is the "test the tests" step. Each mutation → rebuild →
verify test fails → restore source.

/Dependencies/: 2.6, 2.7, 2.8

* Phase 3: Kernel Collection [0/5]

Download, extract, and prepare vendor-patched kernels for the
test matrix. *Highly parallelizable* — each kernel is independent.

** DONE 3.1 Define kernel matrix (=test/kernels/kernels.conf=)
Format: =name|version|boot_method|tier|source=

25 kernels across 3 tiers. Definitive matrix in =research/kernel-matrix.org=,
operational config in =test/kernels/kernels.conf=.

Sources:
- Ubuntu mainline PPA (=ppa:vX.Y=): 24 kernels. Auto-discovered .deb filenames.
  Includes XFS, quota, 9p modules. Version coverage, not vendor patches.
- CentOS vault RPMs (=rpm:<url>=): 2 kernels (centos-6, centos-7).
  Direct URLs to final update kernels. Has XFS + quota, but *no 9p module*.
- Commented out: ubuntu-2604 (7.0, unreleased), mainline-3.0 (not on PPA).

/Verification/: =download.sh --list= shows full matrix with status.

** DONE 3.2 Write kernel download/extract scripts
=test/kernels/download.sh= — reads kernels.conf, downloads + extracts.
Idempotent (skips already-extracted). Records exact URLs in =SOURCE_URLS=
file per kernel for reproducibility.

Features implemented:
- [X] PPA path: scrapes index, auto-discovers .deb files, handles both
  old-style (single linux-image) and new-style (image-unsigned + modules)
- [X] RPM path: =rpm2cpio | cpio= extraction
- [X] Direct .deb path (for future vendor packages)
- [X] Output structure: =test/kernels/<name>/extracted/boot/vmlinuz-*=,
      =test/kernels/<name>/extracted/lib/modules/*/=
- [X] =--list=, =--tier N=, =--kernel NAME=, =--force=, =--clean= options
- [X] =SOURCE_URLS= file per kernel recording exact download URLs

/Dependencies/: 3.1

** DONE 3.3 Download all kernels
All 25 vendor kernels downloaded and extracted (~5.9 GB total).

Switched from mainline PPA kernels to actual distro vendor kernels
(vendor patches affect quota behavior). Only 2 mainline PPA kernels
remain (5.19, 6.2) for Arch/Gentoo coverage where no vendor kernel exists.

Breakdown:
- Tier 1: 10/10 (3 Ubuntu .deb, 3 Debian .deb, 2 Alma RPM, 2 CentOS RPM)
- Tier 2: 8/8 (2 Ubuntu .deb, 2 Debian .deb, 2 openSUSE RPM, 2 Fedora RPM)
- Tier 3: 7/7 (2 Debian .deb, 1 Ubuntu .deb, 1 openSUSE RPM, 1 Amazon RPM, 2 mainline PPA)

Package sources: security.ubuntu.com, security.debian.org,
archive.debian.org, repo.almalinux.org, vault.centos.org,
dl.fedoraproject.org, download.opensuse.org, cdn.amazonlinux.com,
kernel.ubuntu.com/mainline (PPA, only for mainline-5.19 + mainline-6.2).

/Verification/: =download.sh --list= shows 25/25 extracted.

/Dependencies/: 3.2

** DONE 3.4 Build legacy initramfs
Q4 resolved: vng hangs on <4.9, so raw QEMU path is needed.

quotatool itself needs only ONE build for all kernels (see
=research/quotatool-build-analysis.org=): it ships its own quota
structs and does runtime format detection. A normally-linked quotatool
binary built on the host works inside any VM via filesystem sharing.
*No static/musl build needed for quotatool itself.*

If Q4 fails (virtme-ng can't boot old kernels), build a minimal
initramfs containing:
- busybox (statically linked)
- quotatool binary (normal build, shared via 9p or baked in)
- repquota (from matching distro package, see Q2)
- e2fsprogs, xfsprogs, util-linux (mount, losetup)
- test scripts

/Subtasks/ (only if Q4 fails):
- [ ] Gather repquota + deps from distro packages (done in 3.3)
- [ ] Build initramfs generation script
- [ ] Test: boot oldest kernel (3.10) with this initramfs

/Verification/: =qemu-system-x86_64 -kernel <3.10-vmlinuz> -initrd
<initramfs>= boots to a shell with quotatool and quota-tools available.

/Dependencies/: Q4 result, 3.3 (need a kernel to test with)

** DONE 3.5 Boot-verify all kernels
Boot each kernel, run =uname -r=, verify it matches expected version.
=test/kernels/verify-boot.sh= automates this.

Results (2026-03-09):
- *20/20 non-RHEL kernels pass*: all Ubuntu, Debian, Fedora, openSUSE,
  mainline kernels boot and report correct version
- 5 RHEL-derived kernels blocked by Q7 (no 9p module):
  centos-6, centos-7, alma-8, alma-9, amazon-2

Fixes applied during verification:
- virtio_pci.ko added to QEMU initramfs module list
- Module load order fixed (virtio core before virtio_pci)
- Debian 13 usrmerge: modules at =/usr/lib/modules/=, not =/lib/modules/=
- vng command quoting fix for multi-word commands

/Verification/: =verify-boot.sh= shows 20 pass, 5 blocked (Q7).

/Dependencies/: 3.3, 3.4 (for legacy), 1.2

* Phase 4: Multi-Kernel Orchestration [0/4]

The main loop: for each kernel in the matrix, boot it, run full
test suite, collect results, checkpoint.

** DONE 4.1 Build the orchestrator (=test/run-tests=)
Reads kernels.conf, loops over kernels, calls boot layer for each,
collects results, prints summary.

Implemented features:
- [X] Parse kernels.conf
- [X] For each kernel: boot → run test suite → collect output
- [X] Aggregate results: per-kernel and overall summary
- [X] Exit code: 0 if all pass, 1 if any fail
- [X] =--kernel NAME=, =--tier N=, =--host-only=
- [X] =--timeout SECS=, =--verbose=
- [X] Auto-builds quotatool if not present
- [X] Saves per-kernel logs to =test/results/=
- [X] Detects and skips kernels without 9p (Q7)

/Dependencies/: 1.2, 1.3, 2.5, 3.5

** DONE 4.2 Checkpoint/reentrance
Uses =results/<name>.log= files as checkpoints. A kernel is cached
if its log exists and contains =Results:.*0 failed=. Opt-in via
=--only-failed= (re-run only kernels that failed last time). Default
runs clear all logs first so =--only-failed= never finds stale data.

Tested: tier-1 =--only-failed= drops from ~90s to ~5s (9 cached,
1 re-run).

/Dependencies/: 4.1

** DONE 4.3 Parallel execution
Add =run-tests --jobs N= (like =make -jN=) to run N kernels
concurrently using bash background jobs + =wait -n= (bash 4.3+).
No external deps. Default =--jobs 1= (sequential). Verbose forces
=--jobs 1=. Progress counter with =\r= overwrite. Results printed
in matrix order after all complete.

Tested: =--jobs 4= on full 25-kernel matrix — identical pass/fail
results, ~3x faster than sequential.

/Dependencies/: 4.1, 4.2

** DONE 4.4 Full matrix validation
Run complete matrix: all kernels × all tests × both filesystems.

Results (2026-03-10, verified on two hosts):

| Kernel          | Version | Boot      | Tier | ext4  | XFS      | Notes                    |
|-----------------+---------+-----------+------+-------+----------+--------------------------|
| ubuntu-2404     |     6.8 | virtme    |    1 | 16/16 | 15/15+1s | Clean                    |
| ubuntu-2204     |    5.15 | virtme    |    1 | 16/16 | 15/15+1s | Clean                    |
| ubuntu-2004     |     5.4 | virtme    |    1 | 16/16 | 15/15+1s | Clean                    |
| debian-13       |    6.12 | virtme    |    1 | 16/16 | 15/15+1s | Clean                    |
| debian-12       |     6.1 | virtme    |    1 | 16/16 | 15/15+1s | Clean                    |
| debian-11       |    5.10 | virtme    |    1 | 16/16 | 15/15+1s | Clean                    |
| fedora-39       |     6.5 | virtme    |    2 | 16/16 | 15/15+1s | Clean                    |
| opensuse-156    |     6.4 | virtme    |    2 | 16/16 | 15/15+1s | Clean                    |
| mainline-6.2    |     6.2 | virtme    |    3 | 16/16 | 15/15+1s | Clean                    |
| mainline-5.19   |    5.19 | virtme    |    3 | 16/16 | 15/15+1s | Clean                    |
| fedora-30       |     5.0 | qemu+9p   |    2 | 16/16 | 15/15+1s | Clean                    |
| opensuse-152    |     5.3 | qemu+9p   |    3 | 16/16 | 15/15+1s | Clean                    |
| debian-10       |    4.19 | qemu+9p   |    2 | 16/16 | 15/15+1s | Clean                    |
| ubuntu-1804     |    4.15 | qemu+9p   |    2 | 16/16 | 15/15+1s | Clean                    |
| alma-10          |    6.12 | qemu+root |    1 | 16/16 | 16/16    | Clean (rootfs disk)      |
| alma-9           |    5.14 | qemu+root |    1 | 16/16 | 16/16    | Clean (rootfs disk)      |
| alma-8           |    4.18 | qemu+root |    1 | 16/16 | 16/16    | Clean (rootfs disk)      |
| amazon-2        |    4.14 | qemu+root |    3 | 16/16 | 16/16    | QEMU version–dependent   |
| opensuse-151    |    4.12 | qemu+9p   |    2 | 15/16 | 15/15+1s | grace -r fail            |
| debian-9        |     4.9 | qemu+9p   |    2 | 15/16 | 15/15+1s | grace -r fail            |
| ubuntu-1604     |     4.4 | qemu+9p   |    2 | 15/16 | 15/15+1s | grace -r fail            |
| ubuntu-1404     |    3.13 | qemu+9p   |    3 | 15/16 | 15/15+1s | grace -r fail            |
| debian-8        |    3.16 | qemu+9p   |    3 | 15/16 | 15/15+1s | grace -r fail            |
| centos-7         |    3.10 | qemu+root |    1 | 15/16 | 16/16    | grace -r (rootfs disk)   |
| debian-7        |     3.2 | qemu+9p   |    3 | 15/16 | 14/15+1s | grace -r + xfs blkacct   |
| centos-6         |     2.6 | —         |    — |     — | —        | dropped: no 9p, glibc floor |

(+1s = XFS grace period skip, expected)

Verified on two hosts:
- Linux Mint 21.3 (QEMU 6.2)
- Fedora 43 (QEMU 10.1.4)

All failures are known issues (grace -r on ≤4.12, XFS blkacct on
3.2, amazon-2 QEMU version). No unknown regressions.

Known failures:
- *Grace -r restart* (5 kernels, <=4.12): quotatool =-r= flag sets
  grace to 0 instead of restarting. Bug in quotatool, not kernel.
- *XFS block accounting* (debian-7 only): =used=0= after write on
  kernel 3.2 XFS. Old kernel accounting quirk.

Fixes applied during validation:
- ext4/XFS module bundling in QEMU initramfs (loop, ext4+deps, xfs+deps)
- ext4 format fallback: metadata_csum, 64bit stripped for old kernels
- ext4 legacy quota path forced for kernels <4.5 (builtin enforcement broken)
- XFS format fallback: reflink=0 → crc=0 → crc=0+ftype=0
- crc32c_generic softdep for libcrc32c on Debian 8
- Ubuntu linux-modules-extra added (quota_tree.ko)

/Dependencies/: 4.1, 4.2

* Phase 5: CI Wrapper [0/3]                                       :deferred:

Deferred indefinitely. One-person project — local test framework IS
the CI. Run before push. May revisit if contributors appear.

** CANCELLED 5.1 GitHub Actions workflow
=.github/workflows/test.yml= — matrix strategy over kernels.
Each job: checkout → setup KVM → download kernel → run-tests.

/Subtasks/:
- [ ] KVM permission setup (udev rule)
- [ ] Install virtme-ng
- [ ] Kernel caching (actions/cache for downloaded kernels)
- [ ] Matrix definition from kernels.conf
- [ ] TAP output → GitHub Actions summary

/Dependencies/: 4.4

** CANCELLED 5.2 Cirrus CI for FreeBSD
=.cirrus.yml= — FreeBSD VM, build quotatool, test UFS quotas.

/Dependencies/: 4.4 (Linux framework must work first)

** CANCELLED 5.3 CI documentation
Update README with CI badge. Document how to run tests locally.
Document the tested kernel matrix.

/Dependencies/: 5.1, 5.2

* Phase 6: Kernel Matrix Maintenance [0/1]

The kernel matrix is a snapshot. Distros release new kernels
continuously. Without a maintenance strategy, the matrix goes stale
within 12-24 months and the test framework loses its value.

** DONE 6.1 Future-proof the kernel matrix

Design a sustainable process for keeping =kernels.conf= current as
distros release new versions and old ones go EOL. This is not a
one-time task — it's an ongoing concern that needs a low-friction
solution.

Problems to solve:
- *Discovery*: How do we learn that Ubuntu 26.04, AlmaLinux 10,
  Fedora 44, Debian 14, etc. have been released? Manual tracking
  doesn't scale and is easy to forget.
- *URL sourcing*: Each distro has different repository layouts,
  package naming, and mirror structures. URLs break over time
  (packages move from =security.ubuntu.com= to =old-releases=,
  Fedora moves from =releases/= to =archive/=, etc.).
- *Tier rotation*: When a new LTS ships, the old one should move
  from tier 1 → tier 2 → tier 3 → eventually drop. Needs clear
  rules (e.g., "tier 1 = actively supported LTS/stable").
- *Validation*: New kernels may need new mkfs flags, new module
  deps, new boot quirks. Adding a kernel isn't just a URL —
  it needs a test run.
- *Minimum viable process*: Something a single maintainer can do
  in 30 minutes twice a year, not a full-time job.

Possible approaches (not exhaustive, needs design):
- A =kernels.conf= comment convention marking each entry with
  EOL date → simple grep to find stale entries
- A script that checks distro release feeds (RSS, release-monitoring.org,
  endoflife.date API) and flags new releases
- =download.sh --check-urls= to verify existing URLs still resolve
- Document the "add a new kernel" procedure so it's repeatable
- Periodic review cadence (e.g., at each quotatool release)

This is critical infrastructure — the test matrix IS the project's
regression safety net. A stale matrix means untested kernels in
production.

/Dependencies/: 4.4 (matrix must work first)

* Phase 7: Reproducible Cold-Start [4/5]

The test framework must be usable from a fresh =git clone= with
minimal manual steps. Goal: one command after deps are installed.

*Validated 2026-03-10*: clean clone → =./configure && make= →
=run-tests --setup= → full matrix, on three hosts:
- Linux Mint 21.3 (host A): 17 pass, 8 fail (known)
- Fedora 43 (host B): 18 pass, 7 fail (known)
- Linux Mint 21.3 (host C): 17 pass, 8 fail (known)

Priority: this phase comes before CI (Phase 5) and before any
quotatool code changes. A test framework that only works on one
developer's machine is not a test framework.

** DONE 7.1 Replace glibc busybox with musl-static busybox
The initramfs uses a glibc-linked static busybox (=for GNU/Linux 3.2.0=).
This fails on CentOS 6 (kernel 2.6.32) with "kernel too old".

Fix: download official musl-static binary from busybox.net (1.35.0,
1.1MB, 402 applets, SHA256-verified). Zero build dependencies on host.
=build-busybox.sh= downloads with checksum verification, falls back to
building from source with musl-gcc if download fails.

=initramfs/build.sh= prefers local =busybox-musl= over system busybox.

/Results/:
- Busybox init works on kernel 2.6.32 (centos-6) — no "kernel too old"
- debian-8 (3.16) QEMU path: verified passing
- debian-12 (6.1) virtme path: verified passing (no regression)
- centos-6 still blocked: has 9pnet but no 9p filesystem module, and
  rootfs path fails because host glibc (2.35) requires kernel >=3.2.
  centos-6 is EOL since 2020, kernel 2.6.32 — documented as unsupported.

/Dependencies/: none

** DONE 7.2 Write =test/check-deps.sh=
Executable script that validates all required host tools are present.
Replace the static =test/DEPENDENCIES.md= with a runnable check.

/Must check/:
- qemu-system-x86_64 (required)
- virtme-ng / vng (required for kernels >=4.9)
- /dev/kvm accessible (warn if not — tests run 100x slower)
- curl (for downloading busybox and kernels)
- mkfs.ext4 (e2fsprogs)
- mkfs.xfs (xfsprogs)
- quotaon, quotaoff, repquota, quotacheck (quota package)
- losetup, mount (util-linux)
- rpm2cpio + cpio (for RPM kernel extraction)
- dpkg-deb (for DEB kernel extraction)
- curl or wget (for kernel downloads)

/Output/: clear pass/fail per tool, install hints per distro
(Ubuntu/Debian apt, Fedora/RHEL dnf). Exit 0 only if all required.

/Verification/: passes on correctly set up host, fails with helpful
messages on a bare system.

/Dependencies/: none

** DONE 7.3 Self-bootstrapping =run-tests=
Make the orchestrator self-sufficient. Currently requires manual
invocation of download.sh, build-rootfs.sh, initramfs/build.sh.

/Behavior/:
- Run =check-deps.sh= first. Abort with message if deps missing.
- If kernels not downloaded: prompt (or with =--setup= flag, run
  =download.sh= automatically).
- If initramfs not built: build it (fast, <5s).
- If rootfs.img missing or stale (older than quotatool binary):
  rebuild it automatically.
- If quotatool not built: build it.
- Then proceed with test execution.

This means =./test/run-tests --setup= is the single command that
goes from bare clone to full test results (after deps installed).

/Verification/: fresh clone → install deps → =run-tests --setup=
→ all 25 kernels tested, results printed.

/Dependencies/: 7.1, 7.2

** DONE 7.4 Smoke test / setup verification
A =--verify-setup= or =--smoke= flag that quickly validates the
infrastructure works before committing to a full 50-minute run.

/Must test/:
- Boot one virtme kernel (modern path)
- Boot one QEMU+9p kernel (legacy path)
- Boot one QEMU+rootfs kernel (RHEL path)
- Run 1-2 tests on each, confirm pass
- Total time: <30 seconds

/Dependencies/: 7.3

** DONE 7.5 Add EOL dates to =kernels.conf=
Added EOL dates as comments on each entry (standard support, not
ESM/extended). Sources: vendor lifecycle pages + endoflife.date.
Active distros marked with approximate EOL.

Kept as comments rather than a format extension — simpler, no
tooling changes needed, serves the informational purpose.

The originally planned =--check-urls= and =--check-eol= validation
scripts are deferred — nice-to-have, not blocking.

Document the "add a new kernel" procedure step by step.

/Dependencies/: 6.1 (design), 7.3

* Phase 8: Alpine Rootfs — Host-Independent Testing [0/6]

*Branch: =m1-alpine-rootfs= (off =m1-testing=)*

** Problem & Motivation

Two issues solved by one rootfs:

1. *RHEL rootfs complexity*: =build-rootfs.sh= copies host binaries +
   shared libs via =ldd= introspection into a disk image. Fragile,
   host-dependent, breaks when paths change (merged-usr, multiarch).

2. *Future glibc kernel floor*: glibc upstream defaults to
   =--enable-kernel=4.4= since 2.40. Current distros (Fedora 43,
   Mint 21) still build with floor=3.2, but future distros may stop
   overriding this. When that happens, old test kernels break with
   =FATAL: kernel too old=. See =research/glibc-kernel-floor.org=.

Alpine rootfs (musl-based) solves both: no host introspection, no
kernel version floor, one image for all kernels that can't use the
host filesystem directly.

** Approach: Option C (Hybrid)

Static musl quotatool (our code, our build) + Alpine minirootfs for
all other tools. Only new test dep: =musl-gcc= (one package).

Goal: *one Alpine rootfs replaces both* the glibc-floor workaround
AND the existing RHEL rootfs path (=build-rootfs.sh=). If Alpine
works for RHEL kernels too, we retire =build-rootfs.sh= entirely.

** Key Design Decisions

- Alpine rootfs downloaded during =--setup=, not shipped in repo.
  Same pattern as busybox and kernels: download + SHA256-verify.
- Pin Alpine *3.23* (latest stable, Dec 2025, EOL Nov 2027). Exact
  point release + SHA256 hardcoded in script. Bump when needed.
- No per-kernel glibc mapping needed. Runtime detection already
  exists: =GLIBC_MIN_KVER= from =file /bin/sh=, compared with
  =sort -V=. Works automatically as glibc raises floor in future.
- Build-time (=build-alpine-rootfs.sh=) works *without sudo*:
  =apk.static --usermode= (apk v3, Alpine 3.23+). No fakeroot needed.
- Run-time (inside VM) is always root — no issue there.
- =runuser= works. Alpine's package pulls =linux-pam= as dep.
  Minimal PAM config (pam_rootok + pam_permit). At =/sbin/runuser=.

** Tasks

*** DONE 8.1 Download and verify Alpine minirootfs
Script: =test/kernels/build-alpine-rootfs.sh=.
Download Alpine 3.23 minirootfs (~3.8 MB) from
=dl-cdn.alpinelinux.org=. SHA256-verify. Extract to staging dir.
Follow same pattern as =build-busybox.sh=.

URL pattern:
=https://dl-cdn.alpinelinux.org/alpine/v3.23/releases/x86_64/alpine-minirootfs-3.23.N-x86_64.tar.gz=

/Verification/: tarball downloaded, checksum passes, extracts to
staging dir, =/bin/sh= present.

*** DONE 8.2 Install packages into Alpine rootfs (without sudo)
Install into staging dir: bash, e2fsprogs, xfsprogs, quota-tools,
util-linux (mount, losetup, runuser).

*Resolved*: =apk.static= from Alpine 3.23 + =--usermode= (apk v3).
No fakeroot, no unshare needed. =apk.static= downloaded and cached
alongside the minirootfs. 56 packages, 21 MB installed.

=runuser= works — Alpine's package pulls =linux-pam= as dep.
Minimal PAM config (pam_rootok + pam_permit) same as old RHEL rootfs.
=runuser= at =/sbin/runuser= (not =/usr/bin/= — Alpine layout).

/Verification/: =mkfs.ext4=, =mkfs.xfs=, =quotacheck=, =repquota=,
=mount=, =losetup=, and user-switch tool all present and executable.

/Dependencies/: 8.1

*** DONE 8.3 Build static musl quotatool
Add build target for =musl-gcc -static= quotatool. This binary goes
into the Alpine rootfs — our code under test, statically linked,
no glibc dependency.

Actual approach (=-I/usr/include= causes glibc contamination):
- Copy kernel UAPI headers (=linux/=, =asm/=, =asm-generic/=) to
  isolated tmpdir. =asm/= path varies: =/usr/include/asm= (Fedora)
  or =/usr/include/$(gcc -dumpmachine)/asm= (Debian multiarch).
- Provide empty =sys/cdefs.h= stub: =src/linux/linux_quota.h:17=
  includes =<sys/cdefs.h>= (glibc-ism) but uses nothing from it.
  *M3 cleanup candidate*: remove the include from quotatool source.
- Reuse =config.h= from normal =./configure=. No configure with
  musl-gcc (its cross-compile check fails).
- =CC=musl-gcc CFLAGS="-static -I$KINCLUDE"=
Test-only — normal users never do this. =build-alpine-rootfs.sh=
handles all of this automatically.

New test deps: =musl-gcc= (=musl-tools= on Debian/Ubuntu,
=musl-gcc musl-libc-static= on Fedora/RHEL, =musl= on Arch).
Not available on openSUSE base repos. =linux-libc-dev= /
=kernel-headers= (kernel UAPI headers, usually already installed).
=check-deps.sh= not yet updated to check (minor TODO).

/Verification/: =file quotatool-musl= shows =statically linked=.
=./quotatool-musl -V= works on host.

/Dependencies/: none (parallel with 8.1/8.2)

*** DONE 8.4 Create Alpine rootfs disk image
Package the staging dir (Alpine + packages + static quotatool +
test scripts) into an ext4 disk image using =mke2fs -d= (no sudo).
Same approach as current =build-rootfs.sh= for the final image
step, but contents are musl-based instead of host-copied.

Include: test scripts (=guest-run-all.sh=, =tests/=, =lib/=),
=/etc/passwd= + =/etc/group= for test user IDs, minimal PAM
config if needed.

/Verification/: image mounts, contains all needed binaries, test
scripts present at expected paths.

/Dependencies/: 8.2, 8.3

*** DONE 8.5 Integrate into boot paths
Wire Alpine rootfs into =boot.sh= and =run-tests=.

Boot path selection:
#+begin_example
kernel >= glibc_floor && has 9p   → qemu+9p (host fs, unchanged)
kernel >= glibc_floor && virtme   → virtme (host fs, unchanged)
kernel < glibc_floor  OR  no 9p  → qemu + alpine rootfs
#+end_example

If Alpine rootfs works for RHEL kernels (no 9p), retire
=build-rootfs.sh= and unify to one rootfs path. If not, keep
both (Alpine for glibc-floor, host rootfs for RHEL).

Done:
- =--setup= calls =build-alpine-rootfs.sh=
- =--list= shows =qemu+alpine= with reason (glibc floor / no 9p)
- =--smoke= tests the Alpine rootfs path
- Auto-rebuild when quotatool binary is newer
- Added =_needs_alpine_rootfs()= and =_below_glibc_floor()= helpers
- Fixed pre-existing cosmetic bug: =--list= now shows correct boot
  path for RHEL kernels (was showing =virtme= from kernels.conf)

TODO (minor): update =check-deps.sh= to check for =musl-gcc=.

/Dependencies/: 8.4

*** DONE 8.6 Validate on modern host
Run full test suite on all kernels from a Fedora 43+ host
(glibc >= 2.40). Specifically verify the 4 glibc-floor kernels:
debian-7 (3.2), centos-7 (3.10), ubuntu-1404 (3.13), debian-8 (3.16).

Also verify RHEL kernels (alma-8, alma-9, alma-10, centos-7) work via
Alpine rootfs — if yes, retire =build-rootfs.sh=.

Compare results with Mint 21 (glibc 2.35) baseline to confirm
no regressions.

/Dependencies/: 8.5

* Phase 9: Dependency Audit & Minimization [3/3]

Completed 2026-03-15. Three-agent parallel audit of all =test/*.sh=
scripts. ~65 external commands cataloged, classified, cross-referenced.

Result: dep stack is already lean. All tools justified. One fix made
(=grep -oP= → POSIX =sed=, 6 instances). No other replacements
warranted — cost exceeds benefit for all candidates.

** DONE 9.1 Audit all external commands used by the framework
Three parallel agents audited:
- Every external command in =test/=, classified essential/replaceable
- Hard-coded paths, distro assumptions, GNU-specific flags
- Replacement feasibility (python3, busybox, bash builtins)

Key findings:
- ~45 essential commands (VM infra, filesystem/quota tools, compression)
- ~20 replaceable but not worth it (text processing, coreutils)
- All hard-coded paths have fallbacks (merged-usr, multiarch)
- =rpm2cpio= already optional with graceful fallback
- =file=, =strings=, =readelf= all have fallbacks
- No gratuitous deps (no jq, getent, etc.)

** DONE 9.2 Replace or eliminate non-essential dependencies
Only actionable finding: =grep -oP= (Perl regex, GNU extension).
6 instances across 5 files, all replaced with POSIX =sed=.
Verified with 15-case regression test.

All other candidates assessed and rejected:
- =curl= → python3 urllib: setup-only, curl is ubiquitous, no gain
- =rpm2cpio= → python3: already optional, already fails gracefully
- =dpkg-deb= → ar+tar: adds complexity, no portability gain
- =file/strings/readelf= → python3 ELF: ~100 LOC for zero gain

** DONE 9.3 Document the minimal dependency set
Existing =check-deps.sh= already separates required vs optional deps
with distro-adaptive install hints. No further changes needed — the
audit confirmed the current dep list is correct and minimal.

* Dependency Graph

#+begin_example
Phase 1 (Foundation):
  1.1 ──→ 1.2 ──→ 1.5
           1.3 ──↗
  1.1 ──→ 1.4 ──↗

Phase 2 (Tests):
  2.1 ──→ 2.3 ──→ 2.4 ──→ 2.5
  2.2 ──↗

Phase 3 (Kernels):           ← parallelizable with Phase 2
  3.1 ──→ 3.2 ──→ 3.3 ──→ 3.4 ──→ 3.5
                              ↑
                             1.2 (boot layer needed for verification)

Phase 4 (Orchestration):
  2.5 + 3.5 ──→ 4.1 ──→ 4.2 ──→ 4.3
                         4.4 ──↗

Phase 5 (CI):
  4.4 ──→ 5.1, 5.2 ──→ 5.3

Phase 7 (Cold-Start):       ← CURRENT PRIORITY
  7.1 ──→ 7.3
  7.2 ──→ 7.3 ──→ 7.4
  6.1 ──→ 7.5

Phase 8 (Host-Independent Rootfs):
  8.1 ──→ 8.2 ──→ 8.4 ──→ 8.5
  8.1 ──→ 8.3 ──→ 8.4

Phase 9 (Dependency Minimization):
  9.1 ──→ 9.2 ──→ 9.3
#+end_example

Key parallelism opportunities:
- *Phase 2 and Phase 3* can run in parallel (test writing vs kernel collection)
- *Phase 3.3* (kernel downloads) is embarrassingly parallel across agents
- *Phase 3.5* (boot verification) is embarrassingly parallel
- *Phase 4.3* (parallel test execution) runs kernels concurrently

* Decisions Log

| Date       | Decision                                      | Rationale                              |
|------------+-----------------------------------------------+----------------------------------------|
| 2026-03-08 | Local-first, CI as thin wrapper               | Independence from vendor lock-in       |
| 2026-03-08 | virtme-ng for ≥5.4, raw QEMU for <5.4         | virtiofs requires kernel 5.4+          |
| 2026-03-08 | Multi-kernel is core, not optional             | Kernel diffs are #1 bug source         |
| 2026-03-08 | Include 3.x/4.x kernels in initial matrix     | Prevent regressions on legacy systems  |
| 2026-03-08 | TAP output format                              | Standard, machine-parseable, simple    |
| 2026-03-08 | Shell scripts for test framework               | Portable, no deps, matches project DNA |
| 2026-03-08 | One quotatool build for all kernels             | Ships own headers, runtime detection   |
| 2026-03-08 | kern_quota_format() is critical test target     | Fragile, last updated ~kernel 4.x era  |

* Open Questions

** DONE Q1: Static linking for quotatool
*Not needed.* Source analysis shows quotatool ships its own quota
headers and does runtime format detection. One normally-linked build
works across all kernel versions. See =research/quotatool-build-analysis.org=.

If task 3.4 (legacy initramfs) is needed, the initramfs tools
(busybox, repquota, e2fsprogs) may need static builds or bundled
libs, but quotatool itself can be shared into the VM via 9p or
virtiofs. musl-cc remains an option for the *tools*, not for quotatool.

Tool note: [[https://github.com/intoli/exodus][exodus]] (GitHub: intoli/exodus)
bundles dynamic binaries with their .so files. Unmaintained (dead
since 2021). Noted for reference only.

/Affects/: 3.4 (only if Q4 fails)

** DONE Q2: Quota-tools (repquota) in legacy VMs
Need =repquota= inside legacy VMs for test verification. Since
quota-tools link against kernel headers and libc, there's risk of
version-dependent behavior. Best approach: extract =repquota= from
the matching distro package for each kernel (same vendor, same era).
This is a one-time parallel job alongside kernel downloads (3.3).

Each agent downloading a kernel also extracts the matching =quota=
package from the same distro repo. E.g., CentOS 7 kernel → CentOS 7
quota RPM → extract =repquota= binary + libs.

/Affects/: 3.3, 3.4

** DONE Q3: XFS on very old kernels
Resolved 2026-03-09. XFS works on all tested kernels (3.2 through 6.19)
with format fallback in =fs-setup.sh=. Issues found:
- Modern =mkfs.xfs= creates formats old kernels can't mount (reflink,
  crc, ftype). Fixed by try-and-fallback: default → reflink=0 → crc=0
  → crc=0+ftype=0.
- debian-7 (3.2): XFS block usage accounting reports 0 after writes
  (kernel-level issue, not quotatool bug). 1 test failure.
- All other kernels: full XFS test suite passes.
- CentOS 7 (3.10) not tested due to Q7 (no 9p), but XFS module exists.

/Affects/: 3.1, resolved

** DONE Q5: XFS grace period display/behavior
Grace period test (=t-grace-period.sh=) is skipped on XFS.
After setting global grace with =-t "1 day"= and exceeding the soft
limit, =quotatool -d= reports =block_grace=0= on XFS (works on ext4).

Investigation needed:
- XFS sets global grace via root's limits with =FS_DQ_TIMER_MASK=
  (=src/linux/quota.c= lines 489-494). Code exists and looks correct.
- =d_btimer= is populated from XFS quota structures (line 315).
- Dump code in =main.c:109=:
  =(unsigned long) quota->block_time ? quota->block_time - now : 0=
  — possible C operator precedence issue?
- Issue #27 (fixed in 1.6.4) was about grace on ext4 generic path
  (missing =IIF_BGRACE= defines). XFS path was not affected by that
  fix — may have its own separate bug.
- Default XFS grace is 7 days (=DQ_BTIMELIMIT= in =xfs_quota.h=).
- Separate code paths: =quota_reset_grace()= vs old =xfs_reset_grace()=.

May be a display bug (grace is working but not reported), or the
=-t= command doesn't reach XFS properly. Test with =xfs_quota= tool
inside VM to compare.

/Affects/: test suite (=t-grace-period.sh= XFS skip)

** WAITING Q6: ARM/aarch64 support
Does quotatool work on ARM? The C code uses quotactl() which is
arch-independent, and the shipped quota structs are likely ABI-stable
across architectures. But it hasn't been tested. Research:
- Any arch-specific code paths in src/?
- Would need QEMU user-mode or cross-compilation to test
- Defer until after M1 is complete on x86_64

/Affects/: future architecture support

** DONE Q4: virtme-ng kernel version boundary
Resolved 2026-03-10. Investigation and iterations:
- =vng --force-9p= works on kernels *>=4.9* (tested 4.9, 4.15)
- Hangs on *<=4.4* (tested 3.12, 4.4) — virtio-serial devices
  not supported by these kernels
- Initial boundary: 4.9 (three-tier: virtme+virtiofs ≥5.4,
  virtme+force-9p 4.9-5.3, raw QEMU <4.9)
- *Revised*: force-9p eliminated. vng's initramfs mounts 9p without
  =msize=, old kernels default to 8KB → hang on Fedora 43.
  Our QEMU initramfs sets =msize=524288=. Final boundary: *5.4*.

*Solution*: two boot paths:
- Kernels >=5.4: virtme-ng with virtiofs
- Kernels <5.4: raw QEMU + custom initramfs (busybox + init)
  with 9p (msize=524288) and automatic kernel module loading

Additional findings:
- Old kernels need legacy quota path (quotacheck+quotaon instead
  of =mkfs.ext4 -O quota=) — fs-setup.sh handles both automatically
- Grace period restart (=-r=) broken on kernels ≤4.12
- =su= fails in chroot under old kernels (PAM issue) — replaced
  with =runuser= in all enforcement tests

/Affects/: 1.2, 3.4 (resolved — raw QEMU path works)

** DONE Q7: RHEL-derived vendor kernels lack 9p filesystem module
Resolved 2026-03-09. Option 1 (virtio-blk disk image) implemented.

=build-rootfs.sh= creates a self-contained ext4 disk image (~57MB)
using =mke2fs -d= (no sudo). Contains host binaries, shared libs,
PAM modules, test scripts, quotatool. Mounted read-only via
virtio-blk, tmpfs overlays for writable areas.

=initramfs/init= auto-detects =/dev/vda= → rootfs mode.
=boot.sh= handles compressed modules (=.ko.xz=, =.ko.zst=).
=run-tests= auto-falls back to rootfs for kernels without 9p.
Exit code via serial marker =QUOTATOOL_EXIT:<code>=.

Results:
- alma-10 (6.12): 32/32 pass
- alma-9 (5.14): 32/32 pass
- alma-8 (4.18): 32/32 pass
- centos-7 (3.10): 31/32 (grace-r bug, known)
- centos-6 (2.6): unsupported — busybox fixed (musl-static), but kernel
  2.6.32 has no 9p filesystem module and host glibc (>=2.35) requires
  kernel >=3.2 for rootfs path. EOL since 2020, dropped from matrix.
- amazon-2 (4.14): passes on Fedora 43 (QEMU 10.x), fails on Mint 21
  (QEMU 6.2) — =virtio.ko= insmod "unknown symbol". Exact QEMU
  version boundary unknown. Tier 3.

/Affects/: resolved for all. centos-6 dropped.

** WAITING Q8: Binary output location (=./quotatool= vs =src/quotatool=)
Currently =make= places the binary at project root (=./quotatool=).
Investigate whether =src/quotatool= is more conventional for autoconf
projects of this type. Compare with similar C tools using
=./configure && make=. If =src/= is better practice, update Makefile
and all scripts that reference the binary path (=run-tests=,
=build-rootfs.sh=, =check-deps.sh=, =guest-run-all.sh=).

/Affects/: build system, test framework paths. Low risk, contained change.
