# go-sieve - SIEVE cache eviction for Go

[![Go Reference](https://pkg.go.dev/badge/github.com/opencoff/go-sieve.svg)](https://pkg.go.dev/github.com/opencoff/go-sieve)
[![Go Report Card](https://goreportcard.com/badge/github.com/opencoff/go-sieve)](https://goreportcard.com/report/github.com/opencoff/go-sieve)
[![Release](https://img.shields.io/github/v/release/opencoff/go-sieve)](https://github.com/opencoff/go-sieve/releases)

A Go implementation of the
[SIEVE](https://yazhuozhang.com/assets/pdf/nsdi24-sieve.pdf) cache eviction
algorithm (NSDI'24, Zhang et al.), **engineered from the ground up for highly
concurrent, read-heavy workloads**. Generic over key and value types.

The read path (`Get()`) is fully **lock-free** — a single atomic load on the
key→index map plus a single atomic bit update on a shared visited bitfield.
No mutex, no pointer chasing, no per-entry allocations, zero GC traffic on
hits. Under concurrent read load this is **13–55x faster** than
`hashicorp/golang-lru` per trace on real-world cache traces (warm cache,
10 cores), and the gap grows with core count: SIEVE's aggregate
throughput improves as cores are added while mutex-bound LRU/ARC
throughput *degrades below its own single-core line*. Under a cold-cache
mixed read/write replay, SIEVE is 2–6x faster. The write path uses a
single short-held mutex and a pre-allocated node pool, so
`Add()`/`Probe()` also avoid per-operation heap allocation in steady state.

If you need a cache that many goroutines hit simultaneously on the read
path — an HTTP response cache, a DNS resolver cache, an authz decision
cache, a hot-path object lookup — this implementation is built for that
shape. On a single goroutine the two are comparable (SIEVE replays most
traces faster sequentially, LRU wins a few); under any concurrency,
Sieve wins decisively.

SIEVE uses a FIFO queue with a roving "hand" pointer: cache hits set a
visited bit (lazy promotion), and eviction scans from the hand clearing
visited bits until it finds an unvisited node (quick demotion). It matches
or exceeds LRU/ARC hit ratios with far less bookkeeping — validated here on
~300M requests from the MSR Cambridge and Meta Storage trace repositories.

## Key Design Elements

**Array-backed indexed list.** [Marc Brooker observed](https://brooker.co.za/blog/2023/12/15/sieve.html)
that SIEVE's mid-list removal prevents a simple circular buffer. Rather than
Tobin Baker's "swap tail into hole" workaround, this implementation uses a
doubly-linked list with `int32` indices into a pre-allocated backing array.
This preserves SIEVE's exact eviction semantics while eliminating all interior
pointers — the GC sees a flat `[]node` with no pointers to trace (for
non-pointer `K`, `V` types).

**xsync.MapOf for concurrent access.** The key→index map uses
[puzpuzpuz/xsync.MapOf](https://github.com/puzpuzpuz/xsync) which stores
`int32` values inline in cache-line-padded buckets — no traced pointers per
entry. `Get()` is fully lock-free; only `Add()`/`Probe()` (on miss) and
`Delete()` take the global mutex.

**Columnar slot state (`slotState`).** Each node's lock and visited
counter are hoisted out of the `node` struct into a separate contiguous
`[]uint64` — one word per slot, laid out as a column alongside the
`[]node` array. This columnar split has three effects:

1. the eviction hand scans `IsVisited` by walking a dense `[]uint64` sequentially —
   8 slots per cache line, hardware-prefetch friendly — without pulling in
   key/val data it doesn't need;
2. `Get()`'s `LockAndMark` writes only to the `slotState` word for that slot,
   so it never dirties the cache line holding the node's key/val — no false
   sharing with concurrent readers of adjacent nodes
3. `[]uint64` contains no pointers, so the GC never traces it, unlike node fields
   that may hold pointer-typed K/V.

Within each word, bit 63 is a spinlock and the low bits are a
saturating visited counter (1 bit at k=1, ⌈log₂(k+1)⌉ bits for higher k).

**Pre-allocated node pool.** All nodes are allocated once at cache creation in
a contiguous array. A bump allocator + intrusive freelist (reusing `node.next`)
provides O(1) alloc/free with zero heap allocations during steady-state operation.

**TOCTOU-safe concurrent writes.** `Add()` and `Probe()` use a double-check
pattern: fast-path `Load()` outside the lock, re-check under `mu.Lock()` before
inserting. This prevents duplicate nodes from concurrent writers racing on the
same key.

## Benchmark Results

Benchmarked against [hashicorp/golang-lru v2.0.7](https://github.com/hashicorp/golang-lru)
(LRU and ARC) on an Apple M4 (10 cores: 4 performance + 6 efficiency):

- macOS (Darwin 25.5.0, arm64)
- `go1.26.3`
- `GOMAXPROCS=10`

Benchmarks live in `bench/` as a separate module to avoid polluting `go.mod`.
Every table below is generated by `bench/cmd/mktables` from the raw
results files — including derived claims, which are computed rather than
hand-counted.

Commands used (no name filter — every benchmark in every package runs):

```
cd bench && make bench    # synthetic comparison, count=3
cd bench && make trace    # trace replay + miss ratio + GC, count=1
cd bench && make sweep SWEEP_PROCS="1 2 4 8 10"   # core-scaling sweep
```

Full raw results: [`bench-results.md`](bench-results.md).

### Reading the parallel ns/op numbers

The single-digit ns/op numbers in the parallel tables below are real but
need context: they are **aggregate throughput**.

Go's `b.RunParallel` distributes `b.N` total operations across
`GOMAXPROCS` goroutines and reports `ns/op = wall_clock / b.N`. When 10
goroutines complete Get()s at 5.8 ns/op aggregate, the system produces
one completed Get every ~5.8 ns; the **per-core latency** is ~58 ns
(5.8 × 10 cores), consistent with two L1/L2-hot atomic operations plus
the surrounding map lookup.

LRU/ARC report ~140–390 ns/op under the same conditions — not because a
single Get is that much slower, but because every Get takes a mutex.
All goroutines serialize through one lock, so adding cores makes
aggregate throughput *worse* (see the core-scaling table below: LRU
goes from 31 ns/op at 1 core to 182 ns/op at 10 — lock handoff costs
grow with contention). On a single goroutine, SIEVE and LRU are in the
same league — the **gap is a concurrency-scaling story**.

After warmup, the working data structures (map buckets, node array,
visited bitfield) are resident in CPU cache — L1/L2 for small traces,
L3 for larger ones. Real workloads with lower temporal locality will
see higher per-core latency, but the relative advantage over
mutex-bound LRU/ARC holds whenever there is any parallelism at all.

### Parallel Micro-Benchmarks (`count=3`, medians)

| Benchmark | Sieve | LRU | ARC |
|-----------|-------|-----|-----|
| `Get_Parallel` | **5.76 ns/op, 0 B** | 144.2 ns/op, 0 B | 144.9 ns/op, 0 B |
| `Add_Parallel` | **89.8 ns/op, 8 B** | 212.0 ns/op, 40 B | 273.3 ns/op, 74 B |
| `Probe_Parallel` | **91.2 ns/op, 8 B** | — | — |
| `Delete_Parallel` | 81.2 ns/op, 0 B | **69.5 ns/op, 0 B** | 88.3 ns/op, 0 B |
| `Mixed_Parallel` (60/30/10) | **71.0 ns/op, 2 B** | 162.2 ns/op, 12 B | 211.8 ns/op, 23 B |
| `Zipf_Get_Parallel` (s=1.01) | **24.7 ns/op** | 127.7 ns/op | 133.1 ns/op |
| `Zipf_Get_Parallel` (s=1.50) | **94.2 ns/op** | 100.4 ns/op | 112.6 ns/op |
| Memory @ 1M fill | **78.4 MB** | 112.4 MB | 112.3 MB |
| `GCImpact` (1M entries) | **4.37 ms/iter**, 18.1 µs pause | 9.46 ms/iter, 22.4 µs | 9.93 ms/iter, 23.5 µs |

`Probe_Parallel` is SIEVE-only — LRU's `PeekOrAdd` and `ContainsOrAdd`
skip recency promotion and are not semantic equivalents. `Delete_Parallel`
is the one micro where SIEVE loses: LRU's single-lock linked-list unlink
edges out SIEVE's slot-state clear (69.5 vs 81.2 ns/op). At extreme
Zipf skew (s=1.50) SIEVE's advantage narrows to ~6%: most requests hit
a handful of keys, so readers contend on the same per-slot word.

### Core scaling (`Get_Parallel`, median ns/op)

| GOMAXPROCS | Sieve | LRU | ARC |
|-----------:|------:|----:|----:|
| 1 | 14.9 | 31.4 | 33.8 |
| 2 | 8.9 | 74.3 | 77.0 |
| 4 | 5.5 | 121.1 | 146.0 |
| 8 | 8.0 | 178.7 | 182.0 |
| 10 | 7.5 | 182.1 | 193.4 |

SIEVE's aggregate throughput improves with cores (the 8–10 core numbers
include the M4's efficiency cores); LRU/ARC throughput **degrades**
with every added core because all readers fight over one mutex — at 10
cores LRU is ~6x slower per completed Get than at 1 core.

## Trace Replay Results

This implementation is benchmarked against real-world cache traces from the
[libCacheSim](https://cachelib.org/) trace repository — 13 MSR Cambridge
enterprise block I/O traces + 5 Meta Storage (Tectonic) block traces
totalling ~300M requests. Each trace was replayed with a cache sized at
10% of unique keys, comparing SIEVE (k=1, k=2, k=3) against
hashicorp/golang-lru (LRU and ARC). Parallel benchmarks walk the trace
as a ring with per-goroutine **staggered start offsets** (see
[`bench/README.md`](bench/README.md)).

### Parallel Get throughput (warm cache, 10 goroutines, ns/op, zero allocs)

SIEVE's lock-free `Get()` is **13–55x faster** than the better of
LRU/ARC on every trace under concurrent read load:

| Trace | SIEVE k=1 | SIEVE k=3 | LRU | ARC |
|-------|---------:|---------:|----:|----:|
| msr_2007/msr_web_2 | **2.24** | 2.05 | 105.7 | 150.4 |
| msr_2007/msr_proj_4 | **2.84** | 3.87 | 157.4 | 220.8 |
| msr_2007/msr_prxy_0 | **3.28** | 3.15 | 106.3 | 133.3 |
| msr_2007/msr_prn_1 | **3.51** | 3.37 | 135.9 | 185.8 |
| meta_storage/block_traces_2 | **4.01** | 4.31 | 129.4 | 211.4 |
| meta_storage/block_traces_1 | **4.03** | 4.34 | 132.5 | 197.9 |
| msr_2007/msr_usr_2 | **4.62** | 4.79 | 209.4 | 268.5 |
| meta_storage/block_traces_4 | **4.88** | 4.87 | 159.2 | 250.5 |
| meta_storage/block_traces_3 | **4.95** | 4.56 | 147.2 | 224.8 |
| meta_storage/block_traces_5 | **5.00** | 4.92 | 152.2 | 222.5 |
| msr_2007/msr_src1_0 | **5.65** | 5.60 | 157.4 | 384.9 |
| msr_2007/msr_usr_1 | **6.39** | 7.73 | 252.6 | 308.0 |
| msr_2007/msr_src1_1 | **7.04** | 7.20 | 165.9 | 276.9 |
| msr_2007/msr_proj_1 | **7.31** | 7.21 | 255.1 | 307.3 |
| msr_2007/msr_proj_0 | **8.15** | 8.55 | 110.4 | 136.9 |
| msr_2007/msr_hm_0 | **9.12** | 9.38 | 117.6 | 141.8 |
| msr_2007/msr_proj_2 | **10.09** | 7.96 | 260.1 | 299.6 |
| msr_2007/msr_prn_0 | **12.20** | 12.34 | 173.2 | 154.2 |

The k=3 saturating counter adds little to no overhead on the read path.
This benchmark pre-warms the cache with a full trace replay, then hammers
`Get()` only — the ideal scenario for a read-heavy cache in steady state.

### Parallel Replay throughput (cold cache, mixed read+write, 10 goroutines, ns/op)

This is `Probe()` for SIEVE and `Get+Add` for LRU/ARC, hammered in
parallel with **no warmup**. It measures the steady-state workload a real
cache faces: reads and writes interleaved, evictions happening live. The
previous table's numbers reflect the best-case read ceiling; this table
reflects what throughput you actually get when your cache is doing work.

| Trace | SIEVE k=1 | SIEVE k=3 | LRU | ARC |
|-------|---------:|---------:|----:|----:|
| msr_2007/msr_prxy_0 | **37.4** | 36.6 | 229.3 | 251.1 |
| msr_2007/msr_hm_0 | **58.7** | 61.0 | 228.9 | 297.9 |
| msr_2007/msr_prn_0 | **62.2** | 67.6 | 251.6 | 322.3 |
| msr_2007/msr_proj_0 | **63.0** | 58.6 | 236.2 | 312.2 |
| msr_2007/msr_usr_1 | **73.6** | 80.4 | 376.8 | 391.2 |
| msr_2007/msr_prn_1 | **90.4** | 91.5 | 351.7 | 390.4 |
| msr_2007/msr_proj_4 | **110.0** | 113.5 | 425.1 | 563.9 |
| msr_2007/msr_proj_1 | **118.6** | 121.3 | 447.0 | 505.6 |
| meta_storage/block_traces_1 | **134.2** | 160.0 | 277.2 | 351.6 |
| meta_storage/block_traces_2 | **137.9** | 179.4 | 300.5 | 374.9 |
| msr_2007/msr_web_2 | **155.5** | 145.4 | 478.4 | 513.7 |
| meta_storage/block_traces_4 | **156.2** | 179.2 | 371.8 | 384.8 |
| meta_storage/block_traces_5 | **157.4** | 178.1 | 336.2 | 377.9 |
| meta_storage/block_traces_3 | **158.4** | 178.6 | 312.7 | 380.4 |
| msr_2007/msr_src1_0 | **219.3** | 200.8 | 628.9 | 588.3 |
| msr_2007/msr_usr_2 | **222.3** | 215.6 | 627.7 | 579.9 |
| msr_2007/msr_proj_2 | **252.3** | 265.0 | 528.3 | 527.5 |
| msr_2007/msr_src1_1 | **274.7** | 274.2 | 652.3 | 633.4 |

Under concurrent cold-cache replay, SIEVE is **2–6x faster** than the
better of LRU/ARC on every trace. The margin is smaller than the warm
table because every miss serializes through the same write mutex in
all three caches — SIEVE's edge here comes entirely from its lock-free
hit path. The best case is `msr_prxy_0` (95% hits, 6.1x); the worst
cases are the high-miss-ratio traces (`msr_src1_1`, 98% misses, 2.3x).

(An earlier revision of this table claimed 18–62x. That number was an
artifact of all goroutines walking the trace from offset 0 in lockstep,
which inflated hit ratios; the staggered-offset harness fixes it.)

### Miss ratio (every trace)

Cache sized at 10% of unique keys. **Bold** = best in row.

| Trace | SIEVE k=1 | SIEVE k=2 | SIEVE k=3 | LRU | ARC |
|-------|----------:|----------:|----------:|----:|----:|
| meta_storage/block_traces_1 | 0.4632 | 0.4651 | 0.4672 | **0.4602** | 0.4667 |
| meta_storage/block_traces_2 | 0.4719 | 0.4743 | 0.4754 | **0.4676** | 0.4755 |
| meta_storage/block_traces_3 | 0.4908 | 0.4928 | 0.4948 | **0.4885** | 0.4947 |
| meta_storage/block_traces_4 | 0.4841 | 0.4870 | 0.4888 | **0.4812** | 0.4887 |
| meta_storage/block_traces_5 | 0.4959 | 0.4984 | 0.4998 | **0.4927** | 0.5003 |
| msr_2007/msr_hm_0 | 0.2991 | 0.3025 | 0.3025 | 0.3188 | **0.2923** |
| msr_2007/msr_prn_0 | 0.2156 | 0.2194 | 0.2208 | 0.2310 | **0.2145** |
| msr_2007/msr_prn_1 | 0.3908 | 0.3837 | **0.3796** | 0.4341 | 0.4148 |
| msr_2007/msr_proj_0 | 0.2537 | 0.2660 | 0.2745 | 0.2375 | **0.2242** |
| msr_2007/msr_proj_1 | 0.6794 | 0.6794 | 0.6794 | 0.7215 | **0.6788** |
| msr_2007/msr_proj_2 | 0.8231 | 0.8231 | 0.8231 | 0.8548 | **0.8125** |
| msr_2007/msr_proj_4 | 0.8463 | 0.8463 | 0.8463 | 0.8140 | **0.7173** |
| msr_2007/msr_prxy_0 | 0.0512 | 0.0572 | 0.0594 | 0.0476 | **0.0468** |
| msr_2007/msr_src1_0 | 0.7845 | 0.7845 | 0.7845 | 0.9132 | **0.7811** |
| msr_2007/msr_src1_1 | 0.7939 | 0.7934 | **0.7934** | 0.8129 | 0.8231 |
| msr_2007/msr_usr_1 | 0.3558 | 0.3558 | 0.3558 | 0.4007 | **0.3513** |
| msr_2007/msr_usr_2 | 0.7216 | 0.7216 | 0.7216 | 0.7533 | **0.7199** |
| msr_2007/msr_web_2 | 0.9786 | 0.9786 | 0.9786 | 0.9929 | **0.9785** |

**Overall best (bold, computed by `mktables`):** ARC has the lowest
miss ratio in 11 of 18 rows, LRU in 5 (all meta_storage), SIEVE
k-variants in 2 — msr_prn_1 (k=3 outright) and msr_src1_1 (k=2 and k=3
tied at 0.7934). SIEVE k=1 is never the overall best.

**Head-to-head, SIEVE k=1 vs LRU** (the typical deployment choice):
SIEVE k=1 has lower miss ratio on 10 of 18 traces, LRU on 8. When
SIEVE is better the margins are large (msr_src1_0: 12.9 points,
msr_usr_1: 4.5, msr_prn_1: 4.3). When LRU is better the margins are
narrow (all 5 meta_storage: 2–3 points each).

**SIEVE's case rests on throughput.** Miss ratios are competitive
(SIEVE is never dramatically worse than LRU), and SIEVE is 2–6x faster
on a cold mixed workload and 13–55x faster on warm reads, with the gap
widening as cores are added. SIEVE k=3 produces the single-best entry
in the entire table on msr_prn_1 (0.3796 vs LRU 0.4341, ARC 0.4148).

### Memory during replay

On the 13.2M-request `meta_storage/block_traces_1` trace (601K-entry cache),
`TestGCPressure` reports:

| Variant | TotalAlloc |
|---------|-----------:|
| **SIEVE k=1** | **135 MB** |
| SIEVE k=3 | 135 MB |
| LRU | 418 MB |
| ARC | 997 MB |

SIEVE allocates **3.1x less** than LRU and **7.4x less** than ARC during
replay — the array-backed node pool and inline-int32 `xsync.MapOf` are
structural wins.

Full per-trace tables (sequential replay ns/op, B/op, miss ratio for every
trace) are in [`bench-results.md`](bench-results.md). Methodology and
trace-loading details are in [`bench/README.md`](bench/README.md).

## Usage

```go
import "github.com/opencoff/go-sieve"

// Create a cache mapping string keys to int values, capacity 1000.
c, err := sieve.New[string, int](1000)
if err != nil {
    log.Fatal(err) // ErrInvalidCapacity or ErrInvalidVisitClamp
}

// Or, for constant arguments, use Must to get a one-liner:
c := sieve.Must(sieve.New[string, int](1000))

c.Add("foo", 42)

if val, ok := c.Get("foo"); ok {
    fmt.Println(val) // 42
}

// Probe inserts only if absent; returns the cached value if present.
val, _, r := c.Probe("foo", 99)
// val == 42, r.Hit() == true

c.Delete("foo")
c.Purge() // reset entire cache
```

### SIEVE-k

`WithVisitClamp(k)` creates a SIEVE-k cache where each entry uses a
saturating counter instead of a single visited bit. An item accessed k+1
times survives k eviction passes before being evicted. `k=1` is equivalent
to classic SIEVE (the default). Use `k=2` or `k=3` for workloads with
repeated access patterns where extra eviction resistance is beneficial.

```go
// Classic SIEVE (k=1, the default)
c, err := sieve.New[string, int](1000)

// SIEVE-k=3: items survive up to 3 eviction passes
c, err := sieve.New[string, int](1000, sieve.WithVisitClamp(3))
```

`New` returns an error for `capacity <= 0` (`ErrInvalidCapacity`) and for
`WithVisitClamp(k)` with `k > sieve.MaxVisitClamp` (7 — `ErrInvalidVisitClamp`).
Clamp values below 1 are silently rounded up to 1.

### Eviction Handling

`Add()` and `Probe()` return the evicted entry (if any) along with a
`CacheResult` bitmask. This allows callers to handle evictions
synchronously without channels, goroutines, or lifecycle management.

```go
c := sieve.Must(sieve.New[string, int](1000))

ev, r := c.Add("foo", 42)
if r.Evicted() {
    cleanupDisk(ev.Key, ev.Val)
}

// CacheResult bitmask:
//   CacheHit   — key was already present (value updated, no eviction)
//   CacheEvict — an entry was evicted to make room (mutually exclusive with CacheHit)
```

`Purge()` and `Delete()` do not report evictions.

### Iteration

`All()` returns a Go 1.23+ `iter.Seq2[K, V]` over the cache contents.
Iteration is weakly consistent and safe to call concurrently with
`Get`/`Add`/`Probe`/`Delete` (including from inside the loop body):

```go
for k, v := range c.All() {
    fmt.Println(k, v)
}
```

- Order is unspecified — it does **not** follow SIEVE FIFO order.
- Entries inserted during the walk may or may not be observed; entries
  evicted or deleted during the walk are skipped.
- Walking the cache does **not** mark entries as visited, so iteration
  does not protect entries from eviction (unlike `Get`).
- No cache lock is held across the loop body, so re-entrant
  `Get`/`Add`/`Delete` calls from inside `range` are safe.

## API

| Function / Method | Description |
|-------------------|-------------|
| `New[K, V](capacity, ...Option) (*Sieve[K,V], error)` | Create a cache with fixed capacity. Returns `ErrInvalidCapacity` or `ErrInvalidVisitClamp` on bad input. |
| `Must[K, V](*Sieve[K,V], error) *Sieve[K,V]` | Helper that panics on error; useful with constant arguments. |
| `Get(key) (V, bool)` | Look up a key (lock-free, zero-alloc). |
| `Add(key, val) (Evicted[K,V], CacheResult)` | Insert or update; returns evicted entry and result bitmask. |
| `Probe(key, val) (V, Evicted[K,V], CacheResult)` | Insert-if-absent; returns cached/inserted value, evicted entry, and result bitmask. |
| `Delete(key) bool` | Remove a key. |
| `Purge()` | Clear the entire cache. |
| `Len() int` | Current number of entries (lock-free atomic load). |
| `Cap() int` | Maximum capacity. |
| `All() iter.Seq2[K, V]` | Weakly-consistent range iterator over current entries; does not mark visited. |

### Options

| Option | Description |
|--------|-------------|
| `WithVisitClamp(k)` | Use k-level saturating counters (default k=1 = classic SIEVE). `k` is capped at `MaxVisitClamp` (7); `k < 1` is silently rounded to 1. |

## Key Constraints

**The zero value of `K` is reserved — do not use it as a key** (`""` for
string keys, `0` for integer keys, the zero struct for struct keys).

This is by design: when an entry is evicted or deleted, its slot's key
is zeroed in place, and the lock-free read path relies on a zeroed key
never matching a live lookup. If you store the zero key, a `Get` racing
with that key's eviction can observe the zeroed slot and return a
spurious hit with a zero value. In single-threaded use the zero key
appears to work, but the contract is: keys must be non-zero.

If your natural key space includes the zero value, wrap it (e.g. offset
integer keys by 1, or prefix string keys).

## GC Note

When `K` or `V` is a pointer type (including `string`, which contains an
internal pointer in Go), the node array will still contain GC-traced pointers.
The GC pressure reduction is most dramatic for scalar key/value types (`int`,
`[16]byte`, fixed-size structs).

## License

BSD-2-Clause. See the source files for the full license text.
