Hyper-Token — A Second Approach to Fair Multilingual Tokenization

Built as an independent second approach

Hyper-Token uses script-aware atomic segmentation — grapheme clusters for Latin/Cyrillic, whole characters for CJK, DFA-based syllables for Indic abugidas — followed by adaptive merge scoring, then benchmarked directly against PU-Tok on identical data.

Verified results

FLORES-200 devtest, 30 languages, held-out split, both tokenizers trained fresh at a matched 24K vocab budget:

• Lower tokens/char than PU-Tok in 22 of 30 languages
• ~12% fewer tokens than PU-Tok overall across all 30 languages
• 100% round-trip accuracy across every language tested — zero decode failures
• Trained in 66s vs. PU-Tok's 134s at the same vocab budget
• 2.6M+ characters/second encoding speed, measured on real FLORES-200 text

What we didn't claim

Early internal numbers (50.1% fewer tokens vs. BPE, 1.6M chars/sec) came from a 20-sentence toy corpus, not FLORES-200. We don't publish those anymore — the verified numbers above replace them, and the speed number turned out to be better than the original estimate.