JSON Token Counter

Why Token Counting Matters

Large Language Models like GPT-4 and Claude charge by the token, not by character or word. Understanding how your JSON translates to tokens helps you:

• Stay within context limits — GPT-4 has 8K-128K token limits
• Reduce API costs — Fewer tokens = lower bills
• Optimize prompts — More room for your actual question
• Improve response quality — Less noise in the context

What is a Token?

Tokens are the basic units that LLMs process. They're not quite characters and not quite words — they're somewhere in between:

• "hello" = 1 token
• "Hello, world!" = 4 tokens
• {"name": "John"} = ~7 tokens

JSON tends to use more tokens per character than plain English because of all the punctuation ({} [] : , "").

Token Limits by Model

Optimizing JSON for Tokens

1. Minify Your JSON

Removing whitespace typically saves 10-30% of tokens:

// Before: ~50 tokens
{
  "user": {
    "name": "Alice",
    "email": "alice@example.com"
  }
}

// After: ~35 tokens
{"user":{"name":"Alice","email":"alice@example.com"}}

Use our JSON Minify tool to compress your JSON.

2. Shorten Key Names

Long, descriptive keys are great for readability but costly for tokens:

// Before
{"firstName": "Alice", "lastName": "Smith", "emailAddress": "alice@example.com"}

// After (saves ~30% tokens)
{"fn": "Alice", "ln": "Smith", "email": "alice@example.com"}

Consider using a key mapping in your prompt to maintain clarity.

3. Remove Null/Empty Values

Null values and empty strings still cost tokens:

// Before
{"name": "Alice", "middleName": null, "nickname": ""}

// After
{"name": "Alice"}

4. Use Arrays for Repeated Structures

When you have many similar objects, consider a more compact format:

// Before: Array of objects
[{"name": "Alice", "age": 30}, {"name": "Bob", "age": 25}]

// After: Separate arrays (fewer repeated keys)
{"names": ["Alice", "Bob"], "ages": [30, 25]}

5. Consider Alternative Formats

For very large datasets, consider:

• CSV format — Much more token-efficient for tabular data
• YAML — Slightly more efficient than JSON
• Custom formats — Define your own compact syntax

Programmatic Token Counting

JavaScript/TypeScript

// Using tiktoken (official OpenAI tokenizer)
import { encoding_for_model } from 'tiktoken';

const encoder = encoding_for_model('gpt-4');
const tokens = encoder.encode(JSON.stringify(data));
console.log('Token count:', tokens.length);
encoder.free(); // Don't forget to free memory

Python

import tiktoken
import json

encoder = tiktoken.encoding_for_model("gpt-4")
tokens = encoder.encode(json.dumps(data))
print(f"Token count: {len(tokens)}")

Cost Comparison

Here's how token counts affect your API costs (as of 2024):

Related Tools

• JSON Minify — Compress JSON to save tokens
• JSON Repair — Fix malformed LLM JSON output
• JSON Validator — Validate JSON before sending to LLMs
• JSON Pretty Print — Format JSON for readability

Frequently Asked Questions

Why do different models have different token counts?

Each model family uses a different tokenizer. GPT-4 and GPT-3.5 use cl100k_base, while GPT-4o uses the newer o200k_base which is more efficient. Claude uses its own tokenizer with slightly different characteristics.

Is the token count exact?

This tool provides estimates based on the tokenization algorithms. For production use, consider using the official tiktoken library for exact counts. Our estimates are typically within 5-10% of the actual count.

Do whitespace tokens cost money?

Yes! Every token costs the same, whether it's meaningful content or whitespace. That's why minifying JSON can significantly reduce costs for large payloads.

Should I always minify JSON for LLMs?

Not necessarily. For small payloads, the token savings are minimal and formatted JSON may help the model understand the structure better. For large payloads (1000+ tokens), minification is usually worth it.

Common Mistakes & Pro Tips

• Tokens are not words or characters — Tiktoken splits text into sub-word pieces, so token count roughly tracks length but never equals word or character count. A rough rule of thumb is ~4 characters per token for typical English, but JSON's punctuation and structure make it vary, so always count rather than estimate.
• Pick the encoding that matches your model — cl100k_base is used by GPT-3.5/GPT-4 and the text-embedding-3 models, while o200k_base is used by GPT-4o and o-series models. Counts differ between encodings, so selecting the wrong one gives a number that won't match your actual API usage.
• Keys cost tokens on every repetition — In an array of objects, the key names are re-tokenized for every single element, so verbose keys quietly multiply your token bill across large datasets. Shortening keys or sending data as arrays-of-values can meaningfully cut tokens before the content even matters.
• Minify to save tokens — structural whitespace counts — Indentation, newlines, and spaces after colons are all tokenized, so pretty-printed JSON costs more than minified JSON for identical data. Minifying before sending to an LLM is an easy, lossless way to reclaim context-window budget.
• Claude tokenization differs from tiktoken — Tiktoken's cl100k/o200k encodings are OpenAI's; Anthropic's Claude models use a different tokenizer, so these counts are a close approximation for Claude but not exact. For precise Claude budgeting, treat the tiktoken number as a reasonable upper-bound estimate and leave headroom.