[TOK] [EN] [IZATION] : The Root of All Suffering
The Spelling Tragedy
Why can't LLMs spell words? Tokenization splits words into subwords, losing the atomic nature of complete words.
supercalifragilistic → [super][cal][ifrag][ili][stic]
The String Processing Comedy
Simple tasks like string reversal become complex when text is processed as tokens rather than individual characters.
reverse("hello") → [hell][o] → [o][hell] ≠ "olleh"
The Japanese Sonnet
Non-English languages suffer from suboptimal tokenization, leading to poorer performance and understanding.
こんにちは → [こん][に][ち][は] (4 tokens)
The Arithmetic Soliloquy
Numbers are split into tokens based on patterns, making simple math a complex tokenization puzzle.
1234567890 → [12][345][678][90]
The SolidGoldMagikarp Saga
Long compound words can break into unexpected token combinations, causing confusion and errors.
SolidGoldMagikarp → [Solid][Gold][Mag][ik][arp]
The Universal Truth
All ML language model limitations can be traced back to one villain: Tokenization.
[The][Root][Of][All][Suffering] = [Token][ization]