Experiments and implementations · Intermediate

Implementing and checking attention masks

Implement allowed-position attention and verify zero forbidden weights, row normalization, cached offsets, and explicit empty-row handling.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Define one boolean convention
  2. Normalize only allowed scores
  3. Run and inspect each row
  4. Compare an incorrect normalization
  5. Test the cached offset
  6. Verify an external API before reuse
  7. Ask Logeia about this articleCheck your understanding

Define one boolean convention

This model defines True as allowed and False as forbidden. In three-position self-attention, causal visibility permits keys no later than the absolute query position, and key two is additionally hidden as padding. Combine the rules with logical and. Values are two, four, and one hundred; the large forbidden value makes leakage easy to notice. No model or GPU is used, and all scores are zero. Allowed positions therefore receive equal weights, providing a hand-computable reference for the implementation rather than relying on learned scores whose expected behavior would be harder to isolate.

Normalize only allowed scores

The weights function subtracts the maximum allowed score and exponentiates only allowed entries. Forbidden entries contribute zero, and valid exponentials are normalized by their sum. An entirely forbidden row raises ValueError instead of silently generating NaNs or dividing by zero. This is the exercise’s explicit policy, not a universal instruction for every model implementation. A caller might instead avoid computing padded query rows. What matters is a defined contract for an empty reference set and a test that verifies that policy rather than relying on accidental library behavior.

Run and inspect each row

Save the standard-library code as mask.py and run it with Python 3. Query zero sees only value two and returns two; query one averages two and four and returns three; query two also returns three because the hundred-valued padding key remains hidden. Assertions check unit row sums and zero forbidden weights. No network, external data, or persistent service state is involved. Explain why the large value is excluded using both causal order and padding policy. That explanation is stronger than merely observing a plausible-looking output number.

python
import math
def weights(scores, allowed):
    if not any(allowed):
        raise ValueError('empty allowed row')
    m = max(s for s,a in zip(scores,allowed) if a)
    exp = [math.exp(s-m) if a else 0.0 for s,a in zip(scores,allowed)]
    total = sum(exp)
    return [v/total for v in exp]
values = [2.0,4.0,100.0]
for query_pos in range(3):
    allowed = [key_pos <= query_pos and key_pos != 2 for key_pos in range(3)]
    w = weights([0.0]*3, allowed)
    output = sum(a*b for a,b in zip(w,values))
    print(query_pos, allowed, w, output)
    assert math.isclose(sum(w), 1.0)
    assert all(a or v == 0.0 for a,v in zip(allowed,w))
assert weights([0,0,0], [True,True,False]) == [0.5,0.5,0.0]
cached = weights([0.0]*5, [k <= 4 for k in range(5)])
assert all(math.isclose(w,0.2) for w in cached)
try:
    weights([0], [False])
except ValueError:
    print('empty row rejected')
else:
    raise AssertionError('empty row accepted')

Compare an incorrect normalization

An unrestricted softmax of three zeros gives one third to each position. Zeroing only the final weight leaves total mass two thirds and output two instead of three. Implement that incorrect procedure separately and print its allowed-weight sum. Removing a forbidden contribution is not sufficient if normalization remains wrong. Later renormalization can agree in suitable cases, but needs its own empty-row and numerical policy. The reference implementation starts from the allowed set so those assumptions remain explicit and the hand-calculated result has a clear connection to the code.

Test the cached offset

With four cached positions and a new query at absolute position four, keys zero through four are allowed. Treating the single query array’s local row zero as absolute zero wrongly hides keys one through four. The code verifies five equal weights 0.2 for equal scores. Real scores and values differ, but this controlled case isolates allowed range. Left padding and positional representations must follow compatible indexing. Testing only square masks is insufficient for a cached rectangular attention call, where query and key arrays have different lengths and offsets.

Verify an external API before reuse

Change the padding key to position one and tabulate outputs, then draw a rectangular mask for two cached and two new positions. Completion requires explaining absolute query positions, valid keys, row sums, and empty-row policy. Before translating to an external attention API, check boolean meaning, broadcast shape, and interaction with causal settings in that version’s documentation. This Python function is not claimed to reproduce every library kernel. Recheck outputs and, where relevant, gradients on known small inputs rather than assuming that matching parameter names establishes equivalent semantics.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 19

Choose a learning path