Large CSV Files: Where the Browser Stops and What to Use Instead

"How big is too big" for a CSV has three different answers depending on whether the limit is the browser tab, the parser's memory, or the text box you pasted into.

Where this page stops being the right tool

The converter here reads the whole file into a JavaScript string, parses it into an array of arrays, and serialises the result. That is three copies of your data plus JSON's quoting overhead. Consequences, in the order you will hit them:

1. Paste — a textarea of a few megabytes starts losing keystrokes; browsers do not publish a limit and none is worth testing.
2. Output size — a 100k-row file becomes a JSON array several times larger than the input, and the copy-to-clipboard step is where it dies.
3. Parse memory — roughly file size × 4-6 across input string, parsed rows, and output. A 500 MB file is a multi-GB problem.

Practical ceiling for this tool: tens of thousands of rows, comfortably. A file measured in hundreds of megabytes is a streaming job, and pretending otherwise is how an afternoon disappears.

Stream it instead

Every option below holds flat memory regardless of file size, because it never materialises the file:

import csv, json, sys
with open('big.csv', newline='', encoding='utf-8-sig') as fh:
    for row in csv.DictReader(fh):
        json.dump(row, sys.stdout)   # NDJSON out: one object per line, no giant array
        sys.stdout.write('\n')

Writing NDJSON instead of one huge array is the single biggest win — output streams, and wc -l on the result is a row-count check for free. See CSV to JSON for the two shapes and the round-trip traps.

DuckDB: skip the ETL entirely

duckdb -c "SELECT country, count(*) n FROM read_csv('big.csv', auto_detect=true) GROUP BY 1 ORDER BY n DESC LIMIT 20"
duckdb -c "COPY (SELECT * FROM read_csv('big.csv')) TO 'out.parquet' (FORMAT parquet)"

auto_detect samples the file and infers delimiter, quote char, and header. It is right often enough to be a default and wrong in exactly the ways encoding and the BOM describes, so pass types={'zip':'VARCHAR'} for any identifier column — its type inference has the same leading-zero behaviour as every other inferencer. Parquet out is the usual endgame: smaller than the CSV, typed, and readable by every tool downstream.

Sanity-check a file you cannot open

ls -lh big.csv                                   # is this even the file you think it is
head -c 512 big.csv | hexdump -C | head -6       # BOM? encoding? UTF-16? delimiter?
awk 'NR==1{print NF; exit}' FS=, big.csv         # column count of the header
awk -F, '{print NF}' big.csv | sort | uniq -c    # every distinct column count  <- the big one
wc -l big.csv                                    # LINES, not records (quoted newlines)
tail -c 200 big.csv                              # truncated download? ends mid-field?

awk '{print NF}' | sort | uniq -c is the whole diagnostic: one number means the shape is consistent, two numbers means the file contains two different files. Everything else is a detail.

Truncation and sampling

Most "the big file is broken" reports are really "the download was truncated" — check the byte count against the server's Content-Length before analysing contents. When a sample is enough, take it quote-aware:

head -n 1 big.csv > sample.csv           # header
tail -n +2 big.csv | head -n 5000 >> sample.csv   # only safe if no field contains a newline

head -n counts newlines, so if any quoted field contains one — addresses, comments, JSON blobs — your sample's last record is incomplete and your test is measuring a different file. Safe alternative: sed -n '1,5001p' and drop the final record, or read with a real parser and stop at N records.

For the checks that belong in the load script itself — row-count reconciliation, the empty-string-vs-NULL decision, batching INSERTs — see CSV to SQL. For parsers ranked by behaviour rather than speed, see alternatives.