GigaToken: The 1000x Faster Tokenizer Quietly Rewriting AI's Data Pipeline

Tokenization is the unglamorous plumbing of the AI world. Every language model — from GPT-5.6 to DeepSeek V4 — depends on a tokenizer to chop raw text into the discrete units the model actually processes. It happens before training, before inference, before any of the magic. And until now, almost nobody talked about it.

That changed this week when a project called GigaToken landed on Hacker News and rocketed to over 500 points. Its claim? Tokenize text at gigabytes per second — up to 1000x faster than HuggingFace's widely-used tokenizers library, and hundreds of times faster than OpenAI's tiktoken. Both of those are already written in multithreaded Rust. GigaToken is too, but it's doing something fundamentally different.

What Makes GigaToken Different

Most tokenizers today follow a familiar pattern: read text into memory, run byte-pair encoding (BPE) or SentencePiece, emit token IDs. HuggingFace's tokenizers and tiktoken both do this, and they're fast — tens of megabytes per second on a single core. GigaToken rethinks the entire pipeline.

The key insight is that tokenization is embarrassingly parallel. When you're processing a 12 GB text corpus, you don't need to load it all into memory and process it sequentially. GigaToken reads data directly from disk in chunks, tokenizes each chunk in parallel across all available cores, and streams the results out — all in Rust, with minimal Python overhead.

The result is throughput numbers that sound like a typo:

• 24.5 GB/s for GPT-2 tokenization on a 144-core AMD EPYC system
• 8.8 GB/s on an Apple M4 Max (16 cores)
• 6.3 GB/s on an AMD Ryzen 7 9800X3D (16 threads)
• Up to 1,268x faster than HuggingFace tokenizers on the M4 Max

To put that in perspective: tokenizing the entire OpenWebText corpus (11.9 GB) takes roughly half a second on the EPYC system with GigaToken, versus several minutes with HuggingFace's tokenizer.

Why This Actually Matters

It's tempting to dismiss tokenization speed as an engineering footnote. Frontier model training runs cost tens of millions of dollars and take weeks. What does a few minutes of tokenization save?

Quite a lot, actually. Modern AI labs don't tokenize once and move on. They tokenize constantly — for data exploration, for filtering, for evaluation pipelines, for synthetic data generation, for re-training on new data mixes. When you're iterating on a training corpus, shaving tokenization from hours to seconds changes how you work. Engineers can try a new data mix, tokenize it, and start a training run in minutes instead of overnight.

There's also a memory story. GigaToken's streaming approach means you can tokenize datasets that don't fit in RAM — a real problem when your corpus is measured in terabytes, not gigabytes. HuggingFace's tokenizers need to hold the text in memory. GigaToken just streams through it.

Broad Model Support, Drop-In Replacement

One of GigaToken's strongest selling points is its compatibility layer. It works with virtually every major tokenizer in use today:

• GPT-2 / GPT-OSS (OpenAI's open models)
• Llama 3, 3.1, 3.2, 3.3, and Llama 4
• Qwen 2, 2.5, 3, 3.5, and 3.6
• DeepSeek V3, R1, and V4
• GLM 4 and GLM 5
• Phi-4, Mistral, Gemma, Kimi K2, and more

You can use it in compatibility mode with HuggingFace Tokenizers or tiktoken, which means existing code barely needs to change. The tradeoff is that compatibility mode adds some overhead — you get maybe 10-100x speedup instead of the full 1000x. For maximum throughput, GigaToken's native API reads files directly from Rust, bypassing Python entirely.

The Catch: SentencePiece Tokenizers Lag Behind

GigaToken isn't uniformly fast across all tokenizers. The results show a clear divide: BPE-based tokenizers (GPT-2, Llama, Qwen, DeepSeek) hit the spectacular numbers, but SentencePiece-based tokenizers (Gemma, Mistral, CodeLlama, TinyLlama) see much more modest gains — 7x to 22x. That's still a meaningful improvement, but it's not the headline-grabbing 1000x.

The developer notes that SentencePiece tokenizers are "not well optimized" in GigaToken yet, which suggests there's room for improvement. But the BPE results alone cover the vast majority of today's frontier models.

Implications for the AI Ecosystem

GigaToken arrives at a moment when AI labs are drowning in data. Training runs increasingly involve trillions of tokens, and data pipelines are becoming a competitive bottleneck. If you can tokenize 10x faster, you can iterate on data curation 10x more often — and data quality is arguably the biggest lever in model performance today.

It also levels the playing field. Smaller labs and independent researchers who can't afford massive preprocessing clusters can now tokenize at speeds that were previously the domain of well-funded AI companies with dedicated infrastructure teams.

And there's something satisfying about progress happening at the most fundamental layer of the AI stack. We've gotten used to breakthroughs in model architecture, training algorithms, and reasoning capabilities. But sometimes the biggest wins come from revisiting the boring infrastructure that everything else is built on — and making it 1000x faster.

GigaToken is open source, available on GitHub, and installable via pip. If you're working with language models at any scale, it's worth a serious look.