Goal: define an executable tokenizer contract
This chapter turns an explanatory browser model into a Rust function that accepts real input. The small tokenizer has an explicit contract: return recognized tokens, their original source ranges, and structured diagnostics. Edit HTML in the WASM demo and compare the resulting records with the Rust source. The displayed tree belongs to this teaching implementation. It is not the host browser parsing the input into its own DOM, and it does not imply conformance with the complete HTML parsing standard.
The prerequisites are the earlier HTML token demo, arrays, enums, and functions. Start with four values on paper: the input, a cursor, an output token list, and diagnostics. Ask which operation changes each value. A useful completion criterion is explaining why the cursor advances and why rerunning an identical input produces identical records. A visually plausible page is insufficient evidence because an incorrect tokenizer can still produce something that looks familiar.
Enums and ownership: values that outlive a call
The project uses the closed TokenKind enum with Open, Close, Void, Attribute, and Text variants. Consumers branch on these variants instead of interpreting an input string as a new token category. The Token record keeps the associated name, value, and source range. Treat a range as start inclusive and end exclusive; the range 0..3 covers three bytes. Derive a token length from the endpoints instead of maintaining three independently mutable numbers. This small decision makes assertions about ordering, bounds, and adjacency straightforward to state in tests.
A borrowed string slice can avoid copying, but it cannot remain valid after its owner is dropped. Results crossing into the UI need a clear owner, so distinguish temporary views of input from owned names and text retained in a result. Adding clone everywhere does not replace this reasoning. Document where copying occurs and when input memory may be released. At a foreign function boundary, the other language can violate a pointer contract even when ordinary Rust code would reject the equivalent lifetime mistake.
UTF-8 byte ranges and visible characters
Rust strings use UTF-8, so a visible character need not occupy one byte. In <p>猫</p>, the opening tag occupies 0..3, the text occupies 3..6, and the closing tag occupies 6..10. A cursor written only against ASCII examples may accidentally slice 3..4, which ends inside the encoded character. Searching for ASCII delimiter bytes is compatible with UTF-8, provided the boundaries used to extract text remain valid character boundaries.
The UI selection convention may differ from Rust byte offsets. Keep conversion for display separate from the tokenizer contract, and label any exposed numeric position as bytes. Test an invariant rather than a screenshot: extracting a successful token range from the original input must reproduce the relevant source fragment. Adding Japanese text, an emoji, and a newline should change offsets predictably while leaving delimiter recognition intact. Never repair an offset bug by silently replacing non-ASCII input.
Worked trace: tags, attributes, and text
Trace <p id="x">Hi</p> as a start tag with attributes, text, and an end tag. An angle bracket inside a quoted attribute value cannot automatically end the tag. The scanner needs to know whether it is reading a name, an attribute, or a quoted value. Only after token boundaries are established should tree construction choose a parent. Do not mix a tree-level decision such as being inside a paragraph into the operation that merely identifies the next token.
For a differential experiment, change only the attribute value to "a>b". A scanner that ends the tag at every greater-than byte will incorrectly turn part of the value into text. Then remove a quote and inspect the diagnostic. Decide support according to the project contract rather than assuming a full browser recovery algorithm. A deliberately limited parser can be dependable when its accepted language and rejected cases are explicit. A parser that silently invents meaning for unsupported constructs is harder to test.
| Input fragment | Inspect | Responsibility |
|---|---|---|
| Start tag | Name, attributes, source range | Tokenization |
| Hi | Original text range | Tokenization |
| End tag | Closing name | Tokenization |
| Child of p | Parent node ID | Tree construction |
Report failure as input data
An editable string is expected to contain mistakes. A diagnostic with a position and a reason is a better interface than panicking on every invalid input. For an unterminated quoted value, retain where the value began and report that the input ended before its closing delimiter. Unsupported syntax should also leave an explicit record rather than disappearing. A location is a stable handle for the UI even when an explanation is translated or reworded.
Do not disguise a broken internal invariant as a user syntax error. Invalid input, a resource limit, and an implementation defect are different conditions. On rerun, associate results with the input that actually produced them so an earlier successful output cannot masquerade as the new run. When fixing a diagnostic, first calculate the expected position by hand, then inspect the responsible branch. This avoids a superficial change that matches an error message while leaving the cursor or token boundary incorrect.
Run a check: verify the range unit
The standalone Rust program below checks a minimal contract instead of pretending to test the entire parser. Save it as main.rs, compile with the project’s pinned toolchain, and run it. It extracts the complete Japanese character and verifies that get rejects a range ending inside that character. This establishes the unit of an offset. Passing it is useful evidence about string handling, but it does not prove that the project tokenizer recognizes tags correctly.
Build a test matrix containing ordinary ASCII, empty input, Japanese text, an unterminated quoted attribute, and the documented input limit. Replacing one happy example with ten similar strings does not exercise distinct boundaries. Check ranges and diagnostic presence as well as token counts. If empty input is supported, assert an empty token sequence as a successful result. Keep failure expectations precise enough to distinguish rejection at the intended delimiter from an unrelated crash earlier in the scan.
// Standalone UTF-8 range contract; related to tokenizer source ranges.
fn main() {
let source = "<p>猫</p>";
assert_eq!(source.get(3..6), Some("猫"));
assert_eq!(source.get(3..4), None);
assert_eq!(source.len(), 10);
}Exercise: account for every cursor advance
Read the distributed tokenizer and mark every branch that advances the input position or returns a diagnostic. Run small cases containing only whitespace, a lone opening angle bracket, a lone quote, and Japanese text. If a case does not finish, locate the branch that revisits the same cursor; waiting longer does not establish progress. Short inputs are particularly effective because you can account for every transition without losing track of the state.
For an extension, add exactly one supported construct and a test that distinguishes it from a nearby invalid form. Write down whether the token variants or their payloads must change before editing the scanner. Preserve the old success and rejection tests. Record a passed web check separately from local implementation completion: the latter is your report about code run on your machine, not server grading of that code. Retain the command output and diff so the DOM chapter can separate scanner errors from tree-construction errors.