Parsing Large JSON Files Without Running Out of Memory

A 5 GB JSON file looks harmless until your process gets killed mid-parse. The default tools — JSON.parse, json.load, json.Unmarshal — read the entire document, build the complete object graph in RAM, and only then hand it to you. That model breaks down badly at scale. This article explains why, then walks through streaming approaches in Node.js, Python, Go, and the command line that let you process arbitrarily large files at constant memory.

Why full-parse blows up

When you call JSON.parse(text), two things must coexist in memory: the raw text (or a buffer of it) and the fully materialized object tree. The tree is almost always much larger than the text on disk, and the multiplier is worse than people expect.

Consider an array of records like {"id":123,"value":4.5,"name":"abc"}. On disk that object is around 35 bytes. As a live JavaScript object or Python dict it carries far more weight:

• Per-object overhead. Every object/dict has a header, a hash table or hidden-class pointer, and slots for each key. A small dict in CPython is roughly 200+ bytes before you store any values. A V8 object similarly carries hidden-class and property-backing-store overhead.
• Boxed values. Numbers and booleans become full heap objects (Python int/float, or V8 boxed doubles) instead of packed bytes.
• String duplication. Every record repeats keys like "id" and "name". Without interning you allocate a fresh string per occurrence, so a key that appears a million times can cost a million allocations.

The practical result: a 2 GB JSON file routinely needs 8–10 GB of live heap once parsed. A 5 GB file will OOM on most machines. And because the parse is all-or-nothing, you cannot touch the first record until the last byte is read.

The fix is to never hold the whole thing at once. Read the file as a stream, emit one record at a time, process it, and let it be garbage-collected before the next arrives. Memory stays flat regardless of file size.

The walkthrough: sum a field across a 5 GB array

Here is the task we solve in every language below. The file events.json is a single giant array:

[
  {"id": 1, "type": "click", "amount": 0.5},
  {"id": 2, "type": "view",  "amount": 0.0},
  {"id": 3, "type": "click", "amount": 1.25}
]

We want the sum of amount over all records where type == "click", without ever holding more than one record in memory. The file might be 5 GB and 50 million records — irrelevant once we stream.

NDJSON: the streaming-friendly format

Before the code, understand the format choice. A single big array is awkward to stream because the structure spans the entire file — you cannot split it on newlines, and the opening [ and closing ] bracket the whole thing.

NDJSON (Newline-Delimited JSON, also called JSON Lines / .jsonl) fixes this. It is one complete JSON value per line, with no enclosing array:

{"id": 1, "type": "click", "amount": 0.5}
{"id": 2, "type": "view", "amount": 0.0}
{"id": 3, "type": "click", "amount": 1.25}

Each line is independently parseable. You can read a line, parse it, process it, and discard it. You can split the file into chunks at any newline, count records with wc -l, append new records by appending a line, and process the stream with trivial code. This is why log pipelines, data warehouses (BigQuery, Snowflake), and tools like jq favor NDJSON for bulk data.

Converting between the two is cheap. Array to NDJSON with jq:

jq -c '.[]' events.json > events.ndjson

NDJSON back to a single array:

jq -cs '.' events.ndjson > events.json

The -c flag keeps each output value on one compact line; -s (slurp) collects all inputs into one array. If you control the producer, emit NDJSON in the first place and most of this article's complexity disappears.

Node.js

Streaming a giant array with a SAX-style parser

To stream a single array, you need a parser that emits events as it walks the bytes rather than building the tree. The stream-json package (built on a clarinet-style tokenizer) does exactly this. StreamArray emits one {key, value} per top-level array element:

const fs = require('fs');
const { parser } = require('stream-json');
const { streamArray } = require('stream-json/streamers/StreamArray');

let total = 0;

const pipeline = fs
  .createReadStream('events.json')
  .pipe(parser())
  .pipe(streamArray());

pipeline.on('data', ({ value }) => {
  if (value.type === 'click') total += value.amount;
});

pipeline.on('end', () => {
  console.log('sum of click amounts:', total);
});

Memory stays flat because each record is emitted, handled in the data callback, and then eligible for GC. Node's streams also give you backpressure for free: if your data handler is slow (say it writes to a database), the readable stream pauses reading from disk until the consumer catches up, so the input never outruns the processor and the buffer cannot grow unbounded. If you do async work per record, switch to an async iterator so you can await and let backpressure apply:

const { chain } = require('stream-chain');

const pipeline = chain([
  fs.createReadStream('events.json'),
  parser(),
  streamArray(),
]);

for await (const { value } of pipeline) {
  if (value.type === 'click') total += value.amount;
}

NDJSON with readline

If the file is already NDJSON, you don't need a SAX parser at all — readline hands you one line at a time:

const fs = require('fs');
const readline = require('readline');

const rl = readline.createInterface({
  input: fs.createReadStream('events.ndjson'),
  crlfDelay: Infinity,
});

let total = 0;
for await (const line of rl) {
  if (!line) continue;
  const rec = JSON.parse(line);
  if (rec.type === 'click') total += rec.amount;
}
console.log('sum:', total);

Each JSON.parse operates on a single small line, so peak memory is one record plus the read buffer.

Python

ijson for item-level iteration

The standard json.load(f) materializes everything. ijson instead iterates over events or items. The ijson.items(f, 'item') helper yields each element of the top-level array ('item' is ijson's path token for "each element of an array"):

import ijson

total = 0.0
with open('events.json', 'rb') as f:
    for rec in ijson.items(f, 'item'):
        if rec['type'] == 'click':
            total += rec['amount']

print('sum of click amounts:', total)

Open the file in binary mode ('rb') — ijson works on bytes. For a nested array, say records under a data key, the prefix becomes 'data.item'. Memory is constant: ijson builds each record, yields it, and moves on.

With a C backend installed (ijson.backends.yajl2_c), ijson runs several times faster than the pure-Python tokenizer.

NDJSON with line-by-line json.loads

For NDJSON, plain json.loads per line is both simplest and fastest:

import json

total = 0.0
with open('events.ndjson') as f:
    for line in f:
        line = line.strip()
        if not line:
            continue
        rec = json.loads(line)
        if rec['type'] == 'click':
            total += rec['amount']

print('sum:', total)

Iterating the file object yields lines lazily, so the whole file is never in memory. To go faster, swap in orjson — it parses several times quicker than the stdlib and accepts bytes directly:

import orjson

total = 0.0
with open('events.ndjson', 'rb') as f:
    for line in f:
        if not line.strip():
            continue
        rec = orjson.loads(line)
        if rec['type'] == 'click':
            total += rec['amount']

Go

Go's standard library has the cleanest streaming story of all. json.Decoder reads from an io.Reader and can decode one value at a time. The idiom for a giant top-level array is: use Token() to consume the opening [, loop while dec.More() is true decoding each element into a reusable struct, then consume the closing ]. Memory stays constant because only one element is decoded at a time.

package main

import (
	"encoding/json"
	"fmt"
	"log"
	"os"
)

type Event struct {
	ID     int     `json:"id"`
	Type   string  `json:"type"`
	Amount float64 `json:"amount"`
}

func main() {
	f, err := os.Open("events.json")
	if err != nil {
		log.Fatal(err)
	}
	defer f.Close()

	dec := json.NewDecoder(f)

	// Read the opening '[' so we can stream elements.
	if _, err := dec.Token(); err != nil {
		log.Fatal(err)
	}

	var total float64
	for dec.More() {
		var e Event
		if err := dec.Decode(&e); err != nil {
			log.Fatal(err)
		}
		if e.Type == "click" {
			total += e.Amount
		}
	}

	// Read the closing ']'.
	if _, err := dec.Token(); err != nil {
		log.Fatal(err)
	}

	fmt.Printf("sum of click amounts: %g\n", total)
}

dec.Decode(&e) reads exactly one array element each call, so the decoder never buffers more than the current element plus its read buffer — a 5 GB array fits in a few megabytes of RAM.

For NDJSON in Go, it is even simpler: json.Decoder will decode consecutive whitespace-separated values directly, so you can skip the token dance entirely:

dec := json.NewDecoder(f)
var total float64
for {
	var e Event
	if err := dec.Decode(&e); err == io.EOF {
		break
	} else if err != nil {
		log.Fatal(err)
	}
	if e.Type == "click" {
		total += e.Amount
	}
}

Command line with jq

For one-off processing you often don't need to write any code. jq -c '.[]' streams the elements of an array one per line, and you can compute the sum in a single pass:

jq '[.[] | select(.type == "click") | .amount] | add' events.json

That still builds an array internally. For truly huge or deeply nested documents, use jq --stream, which emits [path, value] pairs as it walks and never materializes the whole tree:

jq -cn --stream '
  fromstream(1 | truncate_stream(inputs))
  | select(.type == "click") | .amount
' events.json

--stream is the last resort when even jq's normal parse won't fit in memory. For NDJSON, jq treats each line as a separate input automatically:

jq -s 'map(select(.type == "click") | .amount) | add' events.ndjson

And to simply count records in an NDJSON file, you don't need jq at all:

wc -l events.ndjson

Memory and throughput, approximately

The numbers below are illustrative approximations for a ~5 GB array of small records — your mileage varies with record shape, language, and hardware.

Approach	Peak memory	Notes
JSON.parse / json.load (full)	~20–40 GB	OOM on most machines
stream-json / ijson (array)	~tens of MB	constant, regardless of file size
Go json.Decoder loop	~few MB	smallest footprint, fastest
jq -c '.[]'	low, bounded	great for ad-hoc work
jq --stream	very low	slowest, for pathological inputs

Streaming trades a little CPU per record for flat memory. On large files that trade is overwhelmingly worth it.

Edge cases to watch

• A single giant object, not an array. Element-streaming assumes a top-level array of independent records. If your file is one enormous object ({"users": {...millions of keys...}}), you can't Decode it record-by-record the same way. Use a true event parser — ijson.kvitems(f, 'users') in Python, stream-json's StreamObject, or jq's --stream — to walk key/value pairs without building the whole object.
• Deeply nested structures. SAX-style parsers track depth; pathologically deep nesting can still grow the path stack and, in some parsers, hit recursion limits. Prefer iterative (non-recursive) parsers and validate input depth if it comes from untrusted sources.
• Big integers losing precision. Streaming doesn't change the fact that JSON numbers beyond 2^53 silently lose precision when parsed into floats. If your records carry 64-bit IDs or financial values, configure your parser to read them as strings or decimals before they corrupt. See JSON Numbers and Precision for the full treatment.
• Mixed or malformed lines in NDJSON. A single bad line can abort a naive loop. Wrap per-line json.loads in a try/except (or check the decode error in Go) so one corrupt record doesn't kill a multi-hour job.

Related Tools & Resources

Tools

• JSON Size Analyzer — Estimate on-disk and in-memory size before you decide whether a file even needs streaming.
• JSON Token Counter — See how many tokens and structural elements a large document contains.
• JSON Minify — Strip whitespace to shrink files and speed up the byte-level reads streaming depends on.
• JSON Validator — Confirm a sample or converted NDJSON file is well-formed before a long batch run.

Learn More

• JSON Numbers and Precision — Why big integers lose precision and how to keep 64-bit values intact while streaming.
• Working with JSON in Python — Deeper coverage of json, ijson, and orjson for Python data work.
• Mastering JSON in JavaScript — Streams, JSON.parse, and Node.js patterns explained in depth.
• How to Open JSON Files — Practical ways to inspect files that are too big for an editor.