Concepts and theorems · Intermediate

Attention masks

Separate causal and padding constraints, place masking before softmax, and account for position offsets when a KV cache is present.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Specify allowed key positions
  2. Combine causal and padding rules
  3. Mask scores before normalization
  4. Check a numerical row
  5. Account for cached positions
  6. Verify interface conventions
  7. Ask Logeia about this articleCheck your understanding

Specify allowed key positions

An attention mask specifies which key positions each query may use. With three-position causal self-attention, query zero sees zero, query one sees zero and one, and query two sees all three. The constraint prevents access to future target tokens even when training processes several positions together. A bidirectional encoder need not use the same constraint. Define the allowed-reference table from the task rather than assuming that every element physically present in an input array should be visible to every query. Position availability and permission to attend are different conditions.

Combine causal and padding rules

Padding aligns sequence lengths in a batch but should not become textual evidence. If the third key is padding, exclude it even when a causal rule would otherwise allow it. A position is valid only when both constraints allow it. Handling padded query rows is a separate issue: the resulting outputs may need exclusion from loss or downstream use. Hiding keys does not automatically mask query-side training loss. Check input formatting, attention visibility, and loss selection independently instead of assuming that one mask controls all three stages of the computation.

Mask scores before normalization

Attention weights commonly result from a softmax over scores. Treating forbidden scores as negative infinity makes their exponentials zero, leaving normalization over allowed positions. Simply zeroing forbidden weights after an unrestricted softmax can leave the remaining weights summing to less than one. Renormalization is possible, but starting with the correct allowed set is easier to reason about. An entirely forbidden row has no ordinary normalized distribution because its denominator is zero. Define a policy for that case rather than hoping that infinities, zeros, or library-specific behavior will silently produce meaningful attention.

Check a numerical row

Take three equal zero scores and forbid the final position. Softmax over allowed positions gives weights(one half, one half, zero). Applying unrestricted softmax first and merely zeroing the last weight gives(one third, one third, zero), whose sum is two thirds. For scalar values two, four, and one hundred, the outputs are three and two respectively. The forbidden value is absent in both expressions, yet incorrect normalization changes the result. This hand calculation is an independent reference for the companion implementation lab and shows why masking location matters even when no visibly forbidden value survives.

Account for cached positions

With four past positions in a KV cache and one new query, the key sequence contains five positions. Query row zero in the small current array corresponds to absolute position four, not absolute zero. It may attend to keys zero through four. Confusing local row number with absolute position wrongly hides most of the cache. For rectangular masks, compare explicit query and key positions. Padding and positional representations must use compatible offsets. A slice from a square lower-triangular matrix is not automatically correct without specifying which absolute positions that slice represents.

Verify interface conventions

Boolean True can mean allowed in one API and blocked in another. PyTorch attention interfaces themselves use different conventions, so consult the documentation for the chosen function and version rather than passing the same boolean array by name alone. Draw masks for three-position causal attention, causal attention with padding, and four cached positions plus one query. Completion means explaining the allowed set, normalized row sums, and empty-row handling. After verifying the implementation lab, connect these constraints to the encoder-decoder comparison to understand why visibility rules depend on the model’s task.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 19

Choose a learning path