Building an Indic-First Tokenizer: Why Standard BPE Fails Non-Latin Scripts

The penalty nobody budgets for

Pick almost any off-the-shelf BPE tokenizer and run the same sentence through it in English, then in Hindi. The Hindi version routinely comes out 2–4x longer in tokens for the same content. That's not a small inefficiency — it's slower inference, worse context-window utilization, and a quiet tax on every non-Latin-script language, because the vocabulary was built on Latin-heavy training corpora in the first place.

That penalty is the reason we built PU-Tok — a tokenizer designed Indic-first rather than Indic-adjacent.

What we changed

Indic Priority Fraction of 0.55 — instead of letting vocabulary allocation fall out of raw frequency statistics (which quietly favors whichever script dominates the training mix), we explicitly reserve the majority of the 200,000-token vocabulary budget for Indic scripts.
A Rust acceleration layer for the parts that actually bottleneck at scale — Viterbi segmentation, trie-based matching, and parallel batch encoding through a sharded LRU cache.
200K vocabulary size, which we didn't pick arbitrarily — it came out of a fertility analysis across 32 languages, checking where returns actually diminish.

languages supported

200K

vocabulary size

test regression suite

What didn't work

Two ideas that looked good on paper didn't survive contact with real data, and we're not smoothing that over here:

Morphology suffix protection — intended to help agglutinative languages — actually caused a Hindi fertility regression before we tracked it down and fixed it back to neutral.
Iterative reallocation — a technique for redistributing vocabulary budget after initial training — turned out to be a no-op on smaller corpora, because the vocabulary was already saturated before reallocation even ran.

Neither made it into a highlight reel. Both are logged in the actual research notes, because a negative result that's documented is more useful to us — and to anyone evaluating this work — than a curated success story.

Where the numbers land

Head-to-head against our own second tokenizer, Hyper-Token, on the FLORES-200 benchmark, PU-Tok comes out ahead specifically on scripts with dedicated handling in its pipeline: Cyrillic, Greek, Armenian, Thai, Lao, Khmer, and Amharic.

Why this matters beyond the benchmark

Tokenizer fertility isn't an academic metric — it's the difference between a model that can hold a genuinely long document in context for a low-resource language versus one that runs out of room three paragraphs in. If you're building or evaluating anything that needs to actually work in Hindi, Tamil, Bengali, or the other languages standard tooling treats as an afterthought, this is the layer worth checking first.

Full technical breakdown, including the complete architecture and current benchmark table, is on the PU-Tok research page. If you're building something that needs this kind of tokenizer — or needs a similar problem solved for a different domain — get in touch.