Fix a small input and define token output
Begin with <p id="intro">Hi</p>. The tokenizer produces a start tag named p with an id attribute, character data Hi, and an end tag named p. It does not yet decide the parent of p or its screen position. Tokenization separates input roles; tree building and layout follow later. The teaching implementation declares a small supported fragment language with explicit tags, quoted attributes, and text. It does not claim that splitting tags implements all of HTML.
Initialize the model with the original input, cursor zero, Data state, an empty work buffer, and an empty output list. A pure transition function maps the same current state and character to the same next state. Rewinding only the cursor is insufficient if a partial attribute value or emitted tokens still belong to the future. Save a complete snapshot, or replay a short input deterministically from its beginning to the selected step. The displayed tokens should be actual outputs of those transitions, not a precomputed final list gradually highlighted by an animation. That distinction becomes testable when the user edits the input or changes a quoted attribute value.
A character means different things in different states
A less-than sign in text can begin a tag. A greater-than sign inside a double-quoted attribute, however, belongs to that attribute value. In <p title="a>b">, stopping at the first greater-than sign would cut the value in half. The demonstration therefore shows both the input cursor and the current state, such as Data, Tag name, or Attribute value. Predict state transitions separately from token emission: changing state does not necessarily complete a token.
| Consumed prefix | Current interpretation | Not yet complete |
|---|---|---|
| <p title="a | Inside a quoted attribute value | Attribute and start tag |
| <p title="a> | The > remains attribute data | Attribute and start tag |
| <p title="a>b" | Attribute value complete | Start tag |
| <p title="a>b"> | Start-tag token emitted | Nothing |
Keep a partial token as explicit state
While reading <p id="intro">, retain the tag name, current attribute name, current value, and completed attributes separately. Finishing the name id does not finish the whole attribute. Saving this work area together with the input position permits deterministic pause and resume. Emit the token only after its supported syntax is complete so the tree builder never consumes a half-built attribute. List unsupported forms, such as duplicate attributes or character references, with examples and explicit diagnostics instead of silently dropping input.
Test a start tag with id="intro" and class="note" to establish when current attribute buffers reset. Forgetting to clear a value buffer after saving the first attribute can turn the next value into a mixture of intro and note. Attribute-separating whitespace differs from whitespace inside quotes: the space in title="one two" belongs to the value. The model limits supported element names to html, head, body, h1, h2, p, div, span, a, and br. An unsupported name should produce a declared diagnostic before a display result is invented. Tests should inspect both successful attributes, not just the number of output tokens, because the count can remain correct while buffer reuse corrupts their values.
Choose a character-token grouping rule
The teaching representation may group Hi into one character token or emit one character at a time. Either choice must allow the same text to be reconstructed. Tests should compare combined content and resulting trees rather than relying only on token count. Preserve whitespace as character data: discarding it changes spaces between words and inline elements. Visual whitespace processing belongs to later rendering rules, not to an arbitrary tokenizer optimization. Record the grouping rule so trace steps remain understandable.
Do not disguise unsupported HTML as ordinary text
Real HTML includes comments, character references, script content, and declarations requiring additional states. Treating every less-than sign inside script as a new tag corrupts script content. The initial teaching implementation reports these as unsupported, with an input position and reason. This is not a prediction that a production browser rejects the same input. When studying HTML recovery later, compare the strict teaching fragment language with the standard parser explicitly. A deliberate limitation is useful only when it is visible and testable.
Diagnostics must not insert the original editable source as active HTML in the host page. Show code and parser-state strings as text. Running the learner input inside the product DOM is different from observing a teaching parser. Safe display also does not substitute for implementing more syntax. An unsupported comment can report its starting offset, observed character, and current state. Reset after the failure and run a valid <p>A</p> case to prove that failed work buffers do not survive. Keeping this test beside the syntax tests checks both state recovery and the boundary between displayed source data and host application behavior.
Test the token contract before debugging the tree
For each small input, define expected token kinds, tag names, attributes, text, and source positions. If a DOM test fails, inspect these tokens first. Correct tokens narrow the investigation to tree construction. Test empty input, a tag without attributes, greater-than inside a quoted value, and an unclosed quote separately. Editing input in the demo must clear previous work buffers and emitted tokens. Otherwise an attribute retained from an earlier run could make identical input produce different results.
Exercise: follow a delimiter inside a quoted value
Step through <p title="x>y">A</p> and identify which greater-than sign completes the start tag. It is the one after the closing quote. The attribute value is x>y and the character data is A. Remove the closing attribute quote and the initial implementation should report an unfinished value. Searching only for the first greater-than sign cannot distinguish these cases. Record current state, partial value, and emitted tokens, and explain why DOM parent relationships have not yet been decided.
As an additional task, extract text from <p>A <span>B</span> C</p> while preserving its spaces. The combined character data should be A B C. Compare a tokenizer that emits one character at a time with one that groups consecutive text; the final text should match even if token counts differ. A count difference alone is not a failure. Do not obtain document text by concatenating tag names and attribute values too. Select character-token content for that question, while still passing start and end tags to the tree builder. The same token stream supports different consumers, and each consumer must state which information it uses and which information it preserves.