Study guides · Intermediate

Tokenization, Self-Attention, and Transformers

Trace token IDs into attention-weighted values while separating positions, masks and architectural variants.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Turn text into token IDs
  2. An embedding is not the numeric token order
  3. Trace one attention calculation
  4. Causal masks and padding masks solve different problems
  5. Keep position information separate from architecture
  6. Calculate scale before claiming performance
  7. Use small checks and separate factual evaluation
  8. Ask Logeia about this articleCheck your understanding

Turn text into token IDs

Tokenization maps a text into units from a particular vocabulary and then into integer identifiers. A token need not be a whole word or a single character: spaces, punctuation, word pieces and byte sequences can participate in the representation. The same text can have different token counts under different vocabularies. Therefore, a Japanese character budget cannot safely be converted using an assumed English word ratio. Count the actual representation used by the intended model when enforcing context or billing budgets. Keep the tokenizer, special-token conventions and model together. Substituting another tokenizer while keeping the model weights does not merely change compression; it changes what the input identifiers mean.

An embedding is not the numeric token order

Vocabulary identifiers are lookup indices, not coordinates of semantic similarity. An embedding layer retrieves a vector associated with each identifier. For sequence length T and embedding width d, the result can be viewed as a T-by-d matrix. This mathematical description does not imply that every vector dimension has one human-readable meaning. Input preparation also includes normalization, special tokens, truncation and padding conventions. Record whether padding is on the left or right and how the model identifies it. A model can receive valid integer tensors yet perform an unintended calculation if those conventions differ from training or from the assumptions of its inference implementation.

Trace one attention calculation

Self-attention derives Query, Key and Value representations from the same input sequence. Standard scaled dot-product attention divides query-key dot products by the square root of the key dimension, applies the relevant mask, normalizes the scores with softmax and combines Values. If two allowed scores are both zero, each receives weight one half. Scalar Values two and six consequently produce four. The weighted objects are Values, not token identifiers or raw scores. Different heads use different learned projections before their outputs are combined. An attention weight describes part of a computation; it should not automatically be presented as a causal explanation of the model’s answer.

Causal masks and padding masks solve different problems

For an autoregressive training setup, a prediction must not obtain an answer by looking at future target positions that will be unavailable during generation. A causal mask restricts that access. A padding mask instead excludes artificial elements introduced to equalize sequence lengths. In a three-position causal example, the first position can attend to the first, the second to the first two, and the third to all three. Write out that permitted-position matrix before using a library API. Boolean mask conventions and additive mask conventions differ between interfaces. Test a tiny example and check that disallowed positions do not contribute, rather than trusting the word “mask” alone.

Keep position information separate from architecture

Position representations, including learned embeddings and rotary constructions, give the model information about order. They are not the same mechanism as a causal mask. A mask can restrict future access while the representation of position remains a separate architectural choice. The original Transformer combines an encoder and decoder, whereas many language models use a decoder-only architecture and other models rely on an encoder. Do not transfer bidirectional encoder assumptions into an autoregressive decoder explanation. Similarly, an implementation accepting a longer sequence does not establish that it uses information at every distance equally well. Length support and evaluated quality are distinct properties that require separate evidence.

Calculate scale before claiming performance

Dense self-attention considers pairs of sequence positions, giving T squared score pairs. Increasing T from one thousand to two thousand increases that count from one million to four million. It does not prove that measured latency or allocated accelerator memory is exactly four times larger. Implementations may avoid materializing an entire intermediate matrix; data movement, other layers and batch shapes also affect the result. In inference, reuse of past Keys and Values changes the work performed at each generation step. Distinguish the size of the mathematical operation, actual memory allocation and observed service latency when explaining an optimization or comparing implementations.

Use small checks and separate factual evaluation

A useful sequence of checks is to count input with the matching tokenizer, inspect tensor shapes, calculate a two-element softmax by hand and test that forbidden positions cannot affect the intended output. Passing those checks validates parts of an implementation, not the truth of every sentence generated by a model. Autoregressive generation computes a distribution conditioned on preceding tokens. Fluency, source support and authorization for external actions remain separate evaluation questions. For a performance comparison, hold model identity, input length, generated length and numeric precision constant where possible. Record throughput together with answer quality so that a faster configuration is not mistaken for an equally capable one.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 19

Choose a learning path