Skip to main content

🎮 Tokenization (Practice)

Companion to the Tokenization Algorithms guide. Implement a mini Byte-Pair Encoding (BPE) merge step. Edit and run the code right here — it executes in your browser.

🐍 Your turn​

Loading editor…

🧠 Challenge yourself​

  1. Run merges until vocab reaches a target size.
  2. How does WordPiece's scoring differ from raw frequency?
  3. Why do subword tokenizers handle rare words better than word-level?

Continue the learning path → Matrix Ops Attention Sparse