What Is a Token?

Lesson challenge：Find token boundaries

Task 1 of 7 · Set the goal

Fixed teaching sentenceCurrent: Rule A · Common phrases

AI assistants enjoy learning Tokens.The sentence stays fixed

Token workbench

Boundary

The same sentence can have different token boundaries. Each block is one token.

Rule A (baseline)HiddenCurrent token countHiddenChange from baselineHidden

This page uses simplified rules; actual models may tokenize the text differently.