#+TITLE: XFS Grace Timer Investigation
#+DATE: 2026-03-16

Reference: M3 task 3.6 in =steps/03-cleanup.org=

* Problem Statement

XFS grace timers were not visible in our test environment. After
investigation, two separate issues found:

1. *XFS lazy writeback* (RESOLVED): grace timer set in memory but not
   visible via quotactl until =sync -f=. Affects all tools equally.
   Fix: added =sync -f= to tests. No quotatool code change.

2. *Grace reset (-r) fails on newer kernels with XFS V4*: after =-r=
   clears the grace timer, user cannot write (EDQUOT) so grace never
   restarts. Only happens with XFS V4 (crc=0) on kernel 6.19+.
   Works fine on kernel 6.8. INVESTIGATION IN PROGRESS.

* Issue 1: XFS Lazy Writeback (RESOLVED)

** Root Cause
XFS updates grace timer in memory on write but does not immediately
flush to on-disk quota metadata. =quotactl= reads return stale data
(timer=0) until synced. =sync -f <mountpoint>= forces the flush.
ext4 updates quota metadata synchronously.

** Fix
Added =sync -f "$MNT"= in =t-grace-period.sh= after writes on XFS.
No quotatool code change needed. Confirmed on loopback and real
block device, kernel 6.8.

* Issue 2: Grace Reset Fails on Newer Kernels (IN PROGRESS)

** Symptoms
=-r= flag uses raise/restore hack: raise limits above usage (clears
grace timer), restore original limits (expects kernel to re-trigger
grace on next write). On some kernel+format combinations, the write
after restore fails with EDQUOT ("Disk quota exceeded") because
the kernel treats timer=0 + over-soft as "grace expired".

** Test Matrix

| Host kernel | XFS format | -r works? | Notes                      |
|-------------+------------+-----------+----------------------------|
| 6.8.0       | V5 (crc=1) | YES       | Default mkfs 5.13          |
| 6.8.0       | V4 (crc=0) | YES       | Forced V4                  |
| 6.19        | V4 (crc=0) | NO        | Native, newer mkfs → V4    |
| VM 5.15     | V5 (crc=1) | YES       | Host with mkfs 5.13        |
| VM 5.15     | V4 (crc=0) | NO        | Host with newer mkfs → V4  |
| VM 5.4      | V4 (crc=0) | YES       | Both host configurations   |

Key observations:
- Kernel 6.8 + V4: works. Kernel 6.19 + V4: fails.
- VM guest 5.4 + V4: works. VM guest 5.15 + V4: fails on one host.
- The VM results on one host are puzzling: guest kernel should handle
  XFS independently of host, yet host environment seems to matter.

** Direct Timer Setting Works
On kernel 6.19, using =xfs_quota= to directly set per-user
timer via =Q_XSETQLIM= with =FS_DQ_TIMER_MASK= works:
- Set timer after clearing grace → timer appears in quotactl
- Write succeeds → grace active
This is a viable fix for =quota_reset_grace()= on XFS.

** Agent Research: No XFS Enforcement Code Changed
Exhaustive search of kernel XFS quota code between v6.8 and v6.18:
the enforcement functions (=xfs_dqresv_check=, =xfs_qm_adjust_res_timer=,
=xfs_trans_dqresv=) are byte-identical. Changes in that range are
limited to dquot buffer lifecycle, LRU reclaim, and debug assertions.

** Block Accounting (50→52) — Working as Intended
XFS rounds quota limits up to its allocation unit (4KB). Normal XFS
behavior, confirmed by user. Not a quotatool issue.

** XFS Image Size
512M image caused "No space left on device" in VMs — root cause was
a stale =/tmp/test-xfs.img= on the host from manual XFS testing.
Virtme's overlay tried to COW the 512M file and overflowed the
upper-layer tmpfs. Deleting the host file fixed it. Size kept at
512M (needed by newer xfsprogs 6.x).

** Direct Timer Set: Kernel Boundary
=Q_XSETQLIM= with =FS_DQ_BTIMER= (direct per-user timer set):
- Works on kernels >=5.10 (timer takes effect immediately)
- Silently ignored on kernels <=5.4 (quotactl returns 0 but no change)
Boundary between 5.4 and 5.10.

* Implementation: Hybrid Approach

=quota_reset_grace()= XFS path now does BOTH:
1. Raise/restore hack (works on old kernels <=5.4 where next write
   re-triggers grace; needed because direct set is ignored)
2. Direct timer set via =Q_XSETQLIM= + =FS_DQ_BTIMER= (works on
   >=5.10; needed because newer kernels 6.19+ block writes after
   grace clear, preventing the hack's re-trigger)

This is safe: on old kernels, step 2 is silently ignored. On new
kernels, step 1 clears the old timer and step 2 sets the new one.

*Verified 25/25 on multiple hosts (2026-03-17).*

** DONE Verify hybrid on multiple hosts
Confirmed: 25/25 on hosts with different QEMU versions, mkfs.xfs
versions, and host kernels. The hybrid approach works universally.

** VM guest kernel mystery (unresolved, low priority)
Guest kernel 5.15 behaves differently on different hosts for XFS
V4 quota. The hybrid fix makes this moot — all tests pass — but
the root cause is unknown. Possibly related to QEMU version
differences or subtle virtiofs/overlay behavior.
