Subword Tokenization

Pick the right-sized pieces. A word vocabulary maps every unseen word to one unknown id. A character vocabulary can spell any word made from known base symbols, but even common words become long sequences. Subword tokenization lands in the middle, so the language model predicts reusable pieces instead of whole words or single characters.

How BPE trains. BPE starts each training word as a list of symbols: signaling becomes [s, i, g, n, a, l, i, n, g]. It counts every neighboring pair, picks the most common one, replaces every non-overlapping match, and then counts again. You have to recount because each merge creates new possible pairs.

Keep pairs distinct. A pair is two separate symbols, not just the spelling you get when you join them. If you made keys with plain string concatenation, [a, bc] and [ab, c] would both turn into abc. The code uses JSON array keys like ["s","i"] so those pairs stay separate and are easy to see while debugging.

Replay merges in order. What BPE learns is an ordered list of merges. To encode new text, start with the same base symbols and try each learned merge once in its training order. For abc, applying [a,b] before [ab,c] produces [abc]. Reverse the order, and you're left with [ab,c].

The system-wide tradeoff. A bigger merge budget usually makes encoded sequences shorter, but it also makes the model's embedding and output matrices larger. Tokenizer design changes context use, parameter count, serving cost, and the exact token ids the trained model expects. It's part of the model contract, not just a harmless preprocessing step.