# The world's fastest CSV parser

Below are benchmark results measuring time and memory usage.
To isolate the raw performance of only the parsing engine, these benchmarks use basic `count` and `select`.

These tests are designed to measure the core CSV parser only, and not various other CLI features--
other tools compared here are often are far better than `zsv` for those, but when it comes just to CSV
parsing, `zsv` is hard to beat.

Note: zsv's `--parallel` feature used in some of these tests is available as of version 1.2.0.
Earlier versions should exhibit the same performance when run single-threaded (without `--parallel`)

## Benchmark result summaries

### Count
<img width="975" height="554" alt="image" src="https://github.com/user-attachments/assets/b61f6366-26ed-4345-910d-ca1c1a495b2d" />

### Select
<img width="975" height="554" alt="image" src="https://github.com/user-attachments/assets/ab189ebe-00c1-49a7-8e4a-90220bbdad76" />

## Setup

- MacOS M3 Pro (note: we also ran on Linux (Intel) and the results were consistent with these. If you experience results on
  any hardware / OS that are materially inconsistent with these, please open an issue)
- `/usr/bin/time -l` used to measure real time, user time, and maximum resident set size 
- Input data: 433MB, \~9.5 million rows file, generated by downloading [worldcitiespop.csv](https://burntsushi.net/stuff/worldcitiespop.csv)
  and concatenating it to itself an additional 2 times
- Compared both single-threaded and multi-threaded performance

## Takeaways
- in real time, `zsv --parallel` was the fastest for both count (>= ~25%) and select (>= ~2x)
  - limited to single-threaded mode, `zsv` was fastest for select (>= ~40% vs xsv, ~2.1x vs polars). `polars` was fastest (by 3.5x!) for count, but see below
    - the select results suggest that for count, `polars` is only parsing rows, and not cells
    - as noted below, polars imposes limitations to the CSV input format that other parsers do not
- in memory footprint, `zsv` and `xsv` are *several orders of magnitude* smaller than DuckDB or Polars:
  - single-threaded: `zsv`'s 1.5MB footprint is 2.7x smaller that `xsv` (4MB), 52x smaller than `duckdb` (76MB) and 324x smaller than `polars` (475MB)
  - this makes `zsv` and `xsv` potentially useful in situations where memory or other runtime constraints (e.g. embedded devices) make
    DuckDB or Python/Polars unavailable

## Handicaps

In each of these tests, `zsv` and `xsv` retain full compatibility of "real-world" CSV.
Polars and DuckDB were given the benefit of some "handicaps", relaxing these compatibility requirements,
so that they could run faster (for DuckDB) and produce correct output (for Polars).

In particular:
    - DuckDB run in no-quote mode assumes no embedded newlines
    - Polars does not "correctly" handle embedded quotes if the cell value does not use opening/closing quotes (e.g. `aaa"aaa`,
      which is generally treated the same as `"aaa""aaa"`). Now, it is true that there is no "official" definition of how `aaa"aaa`
      should be parsed, which is why I put "correctly" in quotes. For this purpose, my definition of correct is quite simply:
      *what Excel does*. Not because I view MSFT as some higher authority, but rather because, objectively,
      Excel has a larger CSV user base-- and has had so over a span of many more years-- than any other application in the world.

### Why these parsers were chosen for the benchmarking

The charts shown here compare zsv with other parsers that were closest in speed: `xsv`, `polars`, and `duckdb`. Others that were
evaluated, but not included in the charts shown here, include `tsv-utils`, `csvcut`/`csvstat`, `mlr` (miller), all of which
are also included in the benchmark script, as well as others that are not included in the benchmark script.

Certainly, it is possible that some other parser should be added to this mix--
if you know of any parsers that your test show to be faster than (or nearly as fast as) `zsv`
(whilst, at the least, properly handling quoted values, preferably also handling embedded commas and newlines,
and ideally also handling all forms of unusual CSV)

# Reproducing these results

## Running the benchmarks

To run these benchmark tests yourself, clone this repositor, cd to app/benchmark and run `make graph`,
or examine [Makefile](../app/benchmark/Makefile) which contains the commands for each tool.

## The Commands

Below are examples of the types of commands that were used to run these tests.
The file "worldcitiespop_big.csv" is generated by downloading
[worldcitiespop.csv](https://burntsushi.net/stuff/worldcitiespop.csv)
and concatenating its non-header data to itself an additional 2 times.

#### zsv
```bash
# Count
zsv count worldcitiespop_big.csv
zsv count --parallel worldcitiespop_big.csv

# Select
zsv select worldcitiespop_big.csv -W -n -- 2 1 3-7
zsv select --parallel worldcitiespop_big.csv -W -n -- 2 1 3-7
```

#### xsv
XSV is single-threaded for these operations.

```bash
# Count
xsv count worldcitiespop_big.csv

# Select
xsv select 2,1,3-7 worldcitiespop_big.csv
```

#### DuckDB

```bash
# Count
duckdb -c "SELECT count(*) FROM 'worldcitiespop_big.csv'"

# Count- ignore quotes
duckdb -c "SELECT count(*) FROM read_csv('worldcitiespop_big.csv',all_varchar=true,quote='')"

# Count- ignore quotes- single-threaded
duckdb -c "SET threads=1; SELECT count(*) FROM read_csv('worldcitiespop_big.csv',all_varchar=true,quote='')"

# Select
duckdb -c "COPY (from read_csv('worldcitiespop_big.csv', all_varchar=true) select #2, #1, #3, #4, #5, #6, #7) TO '/dev/null'")

# Select- ignore quotes
duckdb -c "COPY (from read_csv('worldcitiespop_big.csv', all_varchar=true, quote='') select #2, #1, #3, #4, #5, #6, #7) TO '/dev/null' (quote '')")

# Select- ignore quotes- single-threaded
duckdb -c "SET threads=1; COPY (from read_csv('worldcitiespop_big.csv', all_varchar=true, quote='') select #2, #1, #3, #4, #5, #6, #7) TO '/dev/null' (quote '')")
```

#### Polars

```bash
# Count
python3 -c 'import polars as pl; print(pl.scan_csv("worldcitiespop_big.csv").select(pl.len()).collect().item())'

# Count- single-threaded
POLARS_MAX_THREADS=1 python3 -c 'import polars as pl; print(pl.scan_csv("worldcitiespop_big.csv").select(pl.len()).collect().item())'

# Select
python3 -c 'import polars as pl; pl.scan_csv("worldcitiespop_big.csv", infer_schema=False).select(pl.nth(1, 0, 2, 3, 4, 5, 6)).sink_csv("/dev/null")'

# Select- single-threaded
POLARS_MAX_THREADS=1 python3 -c 'import polars as pl; pl.scan_csv("worldcitiespop_big.csv", infer_schema=False).select(pl.nth(1, 0, 2, 3, 4, 5, 6)).sink_csv("/dev/null")'
```

-----

# Miscellaneous

### DuckDB performance variance

DuckDB performance showed significant variance. For example, below
are outputs from two sets of identical commands each run 3 times, showing 
times that vary by approximately 100x in each set (note: these are different
from the runs used for the graphs, as these use `time` for better precision
vs the others use `/usr/bin/time -l` to also capture memory metrics)

```
duckdb               : real 0m41.699s user 0m3.513s sys 4m27.216s
duckdb               : real 0m0.424s user 0m3.394s sys 0m0.269s
duckdb               : real 0m13.809s user 0m4.436s sys 1m20.095s
duckdb (QUOTE='')    : real 0m0.396s user 0m3.420s sys 0m0.244s
duckdb (QUOTE='')    : real 0m25.438s user 0m4.655s sys 2m45.483s
duckdb (QUOTE='')    : real 0m36.661s user 0m5.880s sys 3m52.661s
```

This variance seems to be present only in the `select` operation, but affected
all of the different invocations of `duckdb` that we used. One possible theory
is that the large RAM requirements (consistently > 1GB RAM) might have been causing
allocation wait time, but `polars` did not have this issue even when it used > 1.5GB RAM.

In any case, in our discussion, we only examine the minimum time results, but the 100x variance is
notable nonetheless both for its magnitude and its frequency (more often than not),
and can be visualized in the below comparison of minimums and averages:

<img width="1080" height="716" alt="image" src="https://github.com/user-attachments/assets/7f34a18c-1eaf-4dbb-9b7c-44f3ef94a44a" />
