Specify a toy tokenizer
The toy encoder starts with individual UTF-8 bytes and applies only two explicitly supplied merges that combine the three bytes for あ. It is neither a frequency-trained BPE tokenizer nor the tokenizer shipped with a particular model. Its tiny rule set makes every boundary decision visible. The resulting five-token count must not be used for a service bill or real model context estimate. Real tokenization requires its own vocabulary, normalization, special-token policy, and model-compatible configuration. This experiment studies a boundary mechanism while deliberately avoiding undocumented claims about an unavailable tokenizer.
Inspect the input representation
The input consists of four code points: A, あ, e, and a combining accent. The last two may display as one accented character but remain two code points in the original string. Their UTF-8 lengths are one, three, one, and two bytes, totaling seven. NFC normalization composes e and the accent into é, giving three code points and six bytes overall. Similar appearance does not imply identical byte or code-point sequences. Decide whether the task requires exact preservation of original input or intentionally normalized text before treating these forms as interchangeable.
Trace the ordered merges
Save the code as tokens.py and run it with Python 3’s standard unicodedata module. The first merge combines the first two bytes of あ; the second combines that pair with the remaining byte. Three pieces become one, reducing seven pieces to five. Hexadecimal output makes the individual byte fragments observable without forcing invalid text decoding. Joining the pieces before decoding recovers the original string exactly. The program downloads no model, makes no network request, and modifies no external data. Cleanup involves only the script if it is no longer needed.
import unicodedata
text = "Aあé"
raw = text.encode('utf-8')
pieces = [bytes([b]) for b in raw]
merges = [(bytes([0xe3]), bytes([0x81])), (bytes([0xe3,0x81]), bytes([0x82]))]
for left, right in merges:
out, i = [], 0
while i < len(pieces):
if i+1 < len(pieces) and pieces[i] == left and pieces[i+1] == right:
out.append(left+right)
i += 2
else:
out.append(pieces[i])
i += 1
pieces = out
assert b''.join(pieces).decode('utf-8') == text
nfc = unicodedata.normalize('NFC', text)
print('codepoints/bytes/toy tokens', len(text), len(raw), len(pieces))
print('toy pieces hex', [p.hex() for p in pieces])
print('NFC codepoints/bytes', len(nfc), len(nfc.encode('utf-8')))
assert (len(text), len(raw), len(pieces)) == (4,7,5)
assert (len(nfc),len(nfc.encode('utf-8'))) == (3,6)
try:
bytes([0xe3]).decode('utf-8')
except UnicodeDecodeError:
print('partial character cannot decode alone')
else:
raise AssertionError('expected incomplete UTF-8')
Decode complete sequences
Decoding the leading UTF-8 byte 0xe3 alone raises UnicodeDecodeError because continuation bytes are missing. This does not mean the original text is corrupt. A fragment of a valid sequence need not be independently valid text. The code verifies that expected exception. Streaming or generation systems need a decoder that can buffer incomplete byte sequences and a clear display unit. Replacing each incomplete fragment with a replacement character can lose information that joining the original bytes would have recovered. Text integrity and transport-fragment boundaries must therefore be handled as separate concerns.
Keep normalization policy explicit
The original and NFC strings share the same あ bytes but differ at the end. A real tokenizer that normalizes input may decode to something different from the original code-point sequence. Here merging is checked on the original sequence, while NFC is displayed separately. Reversing the two merge rules can also change the result. Try that reversal and explain why the complete Japanese character no longer becomes one piece. For a trained tokenizer, rule order or priority is part of its specification, not an incidental detail that can be altered while retaining the same encoded representation.
Report counts with their units
Tabulate code-point count, byte count, piece count, and decoded equality for the original input, NFC form, and reversed-rule experiment. Completion means explaining why displayed characters, Unicode code points, UTF-8 bytes, and tokens are different units. For a real model, use its compatible tokenizer and formatting template and count final input IDs. The toy rules do not establish general Japanese compression efficiency or tokenization quality. After obtaining a justified count, use the context-window material to allocate input and output budgets under the actual model and service constraints.