Fun with Tokenization
beserko As you know, tokenization is the process of converting text into numbers so that it can be processed by machine learning models. The granularity of tokenization can vary—from splitting text into individual characters, to sub-words, to complete words.
Naively, I could tokenize the word “Hi” by simply concatenating the ASCII representations of “H” and “i.” It would technically work, but it’s not very efficient. We could also assign each character a sequential number—“a” as 0, “b” as 1, and so on. Better? Hint: how large is the Unicode character set?
There’s a bigger issue: this level of granularity captures relationships between characters, but not between words or phrases. It’s both too fine-grained and inefficient for most modern language models.
Which brings me to this cool site I found:
This site lets you explore how different LLMs tokenize text—without writing any code or importing libraries. It shows the token breakdown and the resulting token IDs in real time. It’s fun to experiment with. Neat!
Pop Quiz: Do LLMs actually see these tokens?