Compressing CSV: Measured Ratios for gzip, zstd, xz, and What CRLF Really Costs

CSV compresses extremely well — it is repetitive text with a fixed row shape — but the interesting part is how much per codec, and which of the usual recommendations survive a measurement. Everything below is from one run on this machine (Linux/arm64, node v24, standard CLI tools), on a generated file rather than an illustrative one.

The file

120,000 rows × 10 columns, 14,678,593 bytes: id, email, name, city, country, signup, amount, note, tags, updated_at. Contains three quoted fields with embedded commas and semicolons, ISO dates, two-decimal amounts, and repeated city/country values — i.e. what a real export looks like, not a synthetic best case.

id,email,name,city,country,signup,amount,note,tags,updated_at
1,[email protected],Eli Silva,Osaka,JP,2026-02-02,2.01,plain note 1,"tag1;tag2",2026-10-02T01:15:00Z
2,[email protected],Faye Nguyen,Quito,GH,2026-03-03,3.02,plain note 2,"tag2;tag3",2026-10-03T02:15:00Z

Measured

Two findings worth carrying out of this page:

gzip -9 lost to gzip -6. Maximum effort is not maximum compression — the search window and match heuristics trade differently, and on this data the extra effort produced 98,873 bytes more. -6 is the default for a reason; do not add -9 to a pipeline as an optimisation without measuring it.

zstd -1 beat gzip at both ends. Smaller output than gzip -9 while running at a level designed for latency. For anything you compress on the fly — an export endpoint, a log shipper — zstd at a low level is currently the correct default, and the gap is large enough that CPU savings are not the argument any more.

Not measured here, so not claimed: brotli (not installed on this box, and it typically lands near or below zstd on text — measure it on your own data before trusting a blog post's numbers, including this one).

Line endings: raw cost, no compressed cost

The usual advice is "use LF, CRLF wastes a byte per line". At 20,000 rows:

CRLF was 20,001 bytes larger raw (2.7 %) and 303 bytes smaller compressed. A repeated \r before every \n is perfectly predictable, so the compressor charges it nothing. If your CSV is stored compressed or shipped over HTTP with content-encoding, line endings are free — do not break a producer to save a byte the deflate stream already had for free.

The place it is not free is parsers that split on \n and leave \r on the last field. That is a correctness problem, not a size problem, and it is documented on encoding and the BOM.

Storage and transfer formats, by job

Parquet is the honest answer for "large CSV kept forever": the 120k-row file above compresses 15× as text, but a columnar format additionally drops the string repetition and stores id as integers. Compression ratios on CSV are impressive until you notice you are paying to compress 120,000 copies of the same ten header names.

Decompress without loading the file

zcat big.csv.gz | tail -n +2 | head -n 1000000      # stream, never lands on disk
python3 -c "
import gzip, csv, json, sys
with gzip.open('big.csv.gz', 'rt', encoding='utf-8', newline='') as fh:
    for r in csv.DictReader(fh):
        sys.stdout.write(json.dumps(r) + '\n')
"

Measured on the uncompressed 14.7 MB file: csv.DictReader → NDJSON completed 120,000 rows in 0.36 s with 9,508 KB peak RSS — that is under 10 MB of memory for a file 1.5× larger than the whole process budget, because the iterator never holds the file. Same code path with a .gz wrapper costs nothing extra in memory.

Contrast: pasting the same file into a browser converter holds the input string, the parsed array, and the output string simultaneously — typically 4-6× the file size. That is the real ceiling, not the format. Details on large CSV files.

What not to do

• Do not compress a CSV you will grep. zcat file.gz | grep is fine at 15 MB and miserable at 15 GB; a columnar or indexed format exists for that.
• Do not gzip something already compressed. A gzipped .csv.gz is how a 14 MB file becomes 19 MB of nothing.
• Do not assume the server compressed it. Check Content-Encoding on the response; Transfer-Encoding: chunked is not compression, and a "download was 300 MB" report is usually an uncompressed endpoint.
• Do not trust ratio tables, including the one above. Compression is data-shaped: a CSV of random UUIDs compresses maybe 1.3×, a CSV with one repeated column 50×. The commands are three lines; run them on your file.

Related: data cleaning for what to do after you decompress it, and CSV to SQL for loading the result.