Study guides · Advanced

Reading JavaScript in Rust: tokens into syntax trees

Follow a bounded JavaScript parser from source bytes through precedence and syntax trees, including unsupported syntax and useful diagnostics.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Goal: determine program structure before execution
  2. Specify the supported language as a grammar
  3. Track byte positions and token boundaries
  4. Worked trace: precedence becomes tree shape
  5. Separate syntax failures from runtime failures
  6. Runnable Rust: compare two expression trees
  7. Investigate one parser change at a time
  8. Exercise: predict structure before running
  9. Ask Logeia about this articleCheck your understanding

Goal: determine program structure before execution

Wave four connects the existing Rust HTML and CSS pipeline to a small JavaScript interpreter. This lesson stops at the boundary before execution. The parser must turn source text into declarations, expressions, branches, and function bodies that another component can evaluate. Showing JavaScript-looking text in a panel is not evidence that this has happened. Inspect the structure produced by Rust and the location associated with a rejected input.

Start with lex_js and parse_program in examples/rust-browser-wave4. The browser interface forwards input and displays results; it does not evaluate the lesson program with the host JavaScript engine. Native CLI and WASM use the same Rust implementation. Comparing their structured output therefore tests a shared parser contract rather than two unrelated demonstrations.

Specify the supported language as a grammar

The supported subset includes numbers, strings, booleans, null, let and const, assignment, arithmetic and comparisons, if, top-level named functions, arguments, and return. This is an educational language contract. It does not establish compatibility with arbitrary ECMAScript programs or existing websites. Loops, closures, and asynchronous constructs are outside the stated subset. An unsupported form should produce a diagnostic, not silently acquire an invented meaning.

Separate statements from expressions while reading the grammar. In let n = 2 + 3;, the declaration is a statement and its initializer is an expression. A function declaration contains a name, parameter list, and body; a call contains a callee and argument expressions. Similar punctuation does not imply identical structure. Make an explicit support matrix before extending a sample.

Track byte positions and token boundaries

A lexer advances a cursor while distinguishing identifiers, literals, operators, and punctuation. Operators such as =, ==, and === share prefixes. Committing after the first character can change the meaning of the program, so examine how the implementation selects the longest supported token. Characters inside a string literal require a different reading context and must not become operators merely because they resemble punctuation.

Rust source slices use UTF-8 byte boundaries. A string containing Japanese text is a useful boundary case because character counts and byte counts diverge. Diagnostic spans and input limits must use their documented unit consistently. Each HTML, CSS, and JavaScript source is limited to 8192 bytes, with a separate limit on the structured JSON request. Measure the transmitted bytes when testing that boundary.

Worked trace: precedence becomes tree shape

For 2 + 3 * 4, multiplication belongs inside the right operand of addition. Evaluating operators immediately from left to right would incorrectly produce twenty. The parser does not need to know the final answer fourteen; it needs the grammar rule that multiplication binds more tightly. Parenthesizing (2 + 3) * 4 changes the tree so that addition becomes the left operand of multiplication.

The table is a worked reasoning trace, not a copied execution log. Inspect tokens first, sketch the expression tree second, and only then ask the evaluator for a value. Use paired expressions that differ by parentheses. An implementation can accidentally return a correct number for one input while building the wrong structure, so a single final value is weak evidence.

StageInput or structureCheck
Lex2 / + / 3 / * / 4Operators differ from numbers
ParseAdd(2, Mul(3, 4))Multiply is the right subtree
Evaluate2 + 12 = 14Compute from structure
ContrastMul(Add(2, 3), 4)Parentheses change the result to 20

Separate syntax failures from runtime failures

The initializer is missing in let n = ;, so parsing can reject it before any value is computed. Reading an unknown identifier is different: the statement may be structurally valid while evaluation fails later. Determine whether parsing completed before interpreting a diagnostic. Calling every failure a JavaScript error hides the distinction between a missing token and an invalid runtime binding or DOM reference.

Use a short declaration as the normal case. Try empty input, nested parentheses, and source close to the byte limit as boundaries. Use an unclosed parenthesis and an unsupported loop as failures. Every case must terminate with a result or diagnostic. Input and depth limits are resource contracts separate from grammatical meaning; raising them requires understanding recursion and allocation costs.

Runnable Rust: compare two expression trees

This standalone host program checks the relationship between tree shape and arithmetic. Compile it as one Rust file. It is not a replacement for the distributed JavaScript lexer and does not parse source text. Constructing the trees directly isolates the information that a correct parser must provide to an evaluator, making the contract easy to inspect without unrelated interface code.

Expr distinguishes numeric leaves from binary operations. Box permits a recursive tree with a finite type size. The evaluator follows that tree; it never scans a source string looking for operators. After both assertions pass, enter the corresponding expressions in the actual CLI and WASM demo. Explain how the implementation's parsed structure satisfies the same relationship rather than treating the small host test as proof of the complete parser.

rust
#[derive(Debug, PartialEq)]
enum Expr { Num(i64), Add(Box<Expr>, Box<Expr>), Mul(Box<Expr>, Box<Expr>) }
fn eval(e: &Expr) -> i64 {
    match e { Expr::Num(n) => *n, Expr::Add(a,b) => eval(a)+eval(b),
        Expr::Mul(a,b) => eval(a)*eval(b) }
}
fn main() {
    let tree = Expr::Add(Box::new(Expr::Num(2)), Box::new(Expr::Mul(
        Box::new(Expr::Num(3)), Box::new(Expr::Num(4)))));
    assert_eq!(eval(&tree), 14);
    let grouped = Expr::Mul(Box::new(Expr::Add(Box::new(Expr::Num(2)),
        Box::new(Expr::Num(3)))), Box::new(Expr::Num(4)));
    assert_eq!(eval(&grouped), 20);
    println!("precedence=14 grouped=20");
}

Investigate one parser change at a time

Adding an operator usually affects more than tokenization. Tokens, precedence, syntax-tree variants, evaluation, diagnostics, and examples form one connected change. A comparison parsed with the wrong precedence can corrupt a whole condition even when individual operators work. Preserve existing normal cases and prepare a small contrasting input that reveals which expressions the new operator consumes on each side.

Learn the boundaries rather than memorizing function names. State what lex_js establishes and what parse_program promises before evaluation begins. Rust types and error returns reveal who consumes input and who preserves source locations. Select one trace frame and locate the corresponding branch in the Rust source. That connection is more informative than merely observing that an example finishes.

Exercise: predict structure before running

Prepare let score = 2 + 3 * 4; and let score = (2 + 3) * 4;. Draw the initializer tree for each before execution. Then remove the right operand of multiplication and predict the first position where the missing expression becomes certain. Run both valid inputs and the invalid one in the same environment, recording values, structure, and diagnostics separately.

For an extension, move a Japanese string into a conditional and compare byte positions with visible characters. Submit the supported grammar list, predicted trees, observed results, and explanations for disagreements. Choose one unsupported construct and identify the stage that should reject it. A useful parser contract explains accepted structure and rejected boundaries, not only a growing collection of successful examples.

Parse and execute JavaScript in Rust

Loading experiment checkpoint…

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Build your CS foundations · STEP 20

Choose a learning path