Concepts and theorems · Intermediate

Tokens

Distinguish tokens from characters, bytes, and words, and inspect how IDs, normalization, and special tokens affect length and decoding.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Define the model input unit
  2. Separate identifiers from meaning
  3. Trace the processing stages
  4. Count formatting tokens too
  5. Distinguish byte and text boundaries
  6. Make a token count reproducible
  7. Ask Logeia about this articleCheck your understanding

Define the model input unit

A token is a unit in the sequence presented to a model. It need not equal a word: pieces can represent word fragments, whitespace, punctuation, or byte-derived fragments. Boundaries depend on the tokenizer’s vocabulary and processing rules. Different tokenizers may produce different IDs and sequence lengths for the same text. A Japanese character is not necessarily one token either. When reasoning about length limits or token-based accounting, identify the tokenizer paired with the intended model rather than assuming that character count or a generic word count is an exact substitute.

Separate identifiers from meaning

Token IDs identify vocabulary entries; consecutive integers do not imply semantic proximity. An ID commonly selects a row in an embedding table. The same number can refer to a different piece in another vocabulary. A mismatched tokenizer and model may accept a syntactically valid integer sequence while interpreting it incorrectly. Manage vocabulary, special-token IDs, and model configuration as one compatible set. Stored token sequences also require version information to remain interpretable later. Treating IDs as self-describing text or as coordinates in a semantic metric space confuses identification with representation.

Trace the processing stages

A tokenizer may normalize text, perform preliminary splitting, and then apply subword rules. Lowercasing during normalization removes distinctions that decoding alone cannot restore. Unicode strings that look alike can also contain different code-point sequences. Inspect the original text, normalized text, pieces, and IDs separately to locate where a change occurs. Not every tokenizer implements the same stages or normalization policy. Document the actual configuration rather than assuming that any encoder-decoder pair guarantees byte-for-byte recovery of the original input or that all visible boundary changes arise in the subword algorithm itself.

Count formatting tokens too

Special tokens can mark sequence starts, ends, padding, or speaker boundaries. Final model input can therefore be longer than tokenization of the visible body alone. Chat templates also introduce role and message formatting. If an illustrative body contains one hundred tokens and its formatting adds ten, the input length is one hundred ten. Ten is an invented example, not a universal chat overhead. Count the final formatted sequence for the actual configuration, and reserve output capacity separately where the model’s context rules require it. Visible character count does not capture these hidden structural additions.

Distinguish byte and text boundaries

The UTF-8 encoding of the Japanese character あ uses three bytes while representing one Unicode code point. Byte-derived pieces may correspond to only part of that encoding; decoding each piece independently can fail even when decoding the complete sequence succeeds. Combining marks and emoji can further separate displayed characters from code points. The companion lab uses a deliberately specified small tokenizer to expose these boundaries. Its counts illustrate a mechanism and must not be presented as counts from an unspecified real model tokenizer. Identification of the exact encoding process is part of any reproducible result.

Make a token count reproducible

Compare visible text, code-point count, and UTF-8 byte count for ASCII A, Japanese あ, and an e followed by a combining accent. If a real tokenizer is available, inspect its final IDs with special-token handling recorded. Completion means distinguishing word, displayed character, code point, byte, token, and ID. A reported model token count should include tokenizer version, normalization, template, and special-token policy. Continue to the context-window article to determine what happens when the resulting sequence exceeds the allowed input or combined input-output budget.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 19

Choose a learning path