Concepts and theorems · Intermediate

Positional encodings

Use sinusoidal coordinates and two-dimensional rotations to explain sequence order, separating positional representation from guaranteed long-context generalization.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Explain why content is insufficient
  2. Construct sinusoidal coordinates
  3. Separate absolute and relative information
  4. Use a rotation to understand relative offsets
  5. Align positions across batching and caching
  6. Test extrapolation rather than assuming it
  7. Ask Logeia about this articleCheck your understanding

Explain why content is insufficient

Reordering the same words can change a sentence’s meaning. Content-only attention scores lack a direct indication of sequence position without another order-sensitive mechanism. Positional representations provide information about positions or relative offsets. Methods can add vectors to input embeddings, add biases to attention scores, or rotate queries and keys. These are different computations. Do not describe every architecture as merely adding a position vector. Identify where position enters the model and what coordinate or relative relationship it is intended to represent before comparing implementations or making claims about sequence behavior.

Construct sinusoidal coordinates

A common sinusoidal construction pairs sin and cos of p divided by 10000 to the power 2i/d. With dimension four, position zero is(0,1,0,1), and position one is(sin 1,cos 1,sin 0.01,cos 0.01). Different pairs operate at different frequencies. This is one published construction, not a universal choice of base or dimension for every model. Adding it to content embeddings requires compatible shape and scale. It does not replace token IDs with position numbers; content identification and positional information remain separate components that are combined at a specified stage.

Separate absolute and relative information

Absolute position measures location from a sequence origin; relative position measures an offset between two locations. Shifting two tokens together changes their absolute positions but preserves their separation. Tasks and architectures differ in how they use these signals. Learned position tables need a policy for indices beyond their stored rows. A formula evaluable at arbitrary p does not, by itself, guarantee trained-model quality far beyond the observed range. Mathematical representability and statistical generalization are separate issues, so identify both the encoding’s domain and the positions encountered during training or evaluation.

Use a rotation to understand relative offsets

For one two-dimensional pair in rotary position embedding, apply position-dependent rotations to query and key. Their dot product can be rewritten as q transpose times a rotation by the difference in angles times k. If q and k both equal(1,0), this becomes the cosine of that angular difference. Multiple coordinate pairs ordinarily use different frequencies; the whole model is not reduced to a single cosine of distance. The small calculation isolates how relative offsets can enter a score while preserving the distinction between this algebraic mechanism and all other learned content processing.

Align positions across batching and caching

With left padding, clarify how array indices correspond to actual token positions. During cached generation, resetting each new query’s position to zero changes its relationship with past keys. Keep the model’s expected position IDs, attention mask, and cache length consistent. Implementations can differ in padding conventions, so do not invent a replacement numbering policy without checking the interface. A useful verification compares processing a short known sequence all at once with processing it incrementally. That test can expose offset errors before they become obscure failures on much longer inputs.

Test extrapolation rather than assuming it

Write the four-dimensional sinusoidal vectors for positions zero and one, then verify that adding the same angular shift to both rotated vectors preserves their dot product in the toy pair. Completion means distinguishing the representation mechanism from long-document answer quality. When changing positional settings, evaluate across lengths and evidence locations. A larger configured context limit does not establish correct use of distant exceptions or numerical facts. Connect this article with context budgeting and the attention-mask lab so that order information and allowed visibility are checked as distinct parts of the model input computation.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 19

Choose a learning path