Part 4 · 2 chapters · ~18 min

M4: Transpiler

Modern JavaScript rewritten to ES5 on the token stream, covering const and let, arrows with defaults and lexical this, and recursively transpiled template literals, with a v3 source map whose base64 VLQ mappings a browser debugger follows back to the original.

Rewriting, and mapping back

change the dialect, keep the behaviour, keep the debugger

Tokens that keep layout and positions.
const and let become var.
Arrows become functions, with defaults guarded and .bind(this) where needed.
Templates become concatenations, transpiled recursively.
Mappings record a generated position and an original position for each piece.
VLQ: relative numbers packed into base64 digits.

tokens versus an AST

This repo rewrites tokens so every step stays visible. Babel, SWC and esbuild parse to an AST, transform the tree and print it, which is what makes destructuring, classes, async/await, optional chaining and correct block scoping possible. The Architecture course part 1 covers the production build pipeline that uses them. The token approach is honest about its subset, and every rewrite is checked by running the original and the output and comparing the results.

TINY-TRANSPILER: REWRITE AND MAP BACK

an arrow function and a template literal rewritten to ES5, with every output token remembering where it came from

swipe the figure sideways, or tap expand for full screen

1/6

tokens

Tokens, keeping everything: the tokenizer keeps whitespace and comments as tokens so the output preserves the author's layout, and records line and column for each. Template literals are kept whole and split into string and expression parts later.

Building it: tiny-transpiler

Repo: repos/transpiler. Tokenizer, transpiler, source maps and a CLI.

code

// src/transpile.ts: an arrow, end to end
function arrow(pStart: number, pEnd: number, arrowTok: Tok, bodyAt: number): number {
  em.emit('function (', arrowTok);
  const ds = params(pStart, pEnd);                       // names emitted, defaults collected
  em.emit(') {');
  emitDefaults(ds);                                      // if (b === void 0) b = 1;
  let end: number, self: boolean;
  if (toks[bodyAt].value === '{') {                      // block body: copy it through
    end = match(bodyAt); self = usesThis(bodyAt, end);
    range(bodyAt + 1, end); em.emit('}', toks[end]); end++;
  } else {                                               // expression body: wrap in return
    end = exprEnd(bodyAt); self = usesThis(bodyAt, end);
    em.emit(' return ', toks[bodyAt]); range(bodyAt, end); em.emit('; }');
  }
  if (self) em.emit('.bind(this)');                      // lexical this
  return end;
}

code

// src/sourcemap.ts: VLQ in 10 lines
export function vlqEncode(n: number): string {
  let v = n < 0 ? ((-n) << 1) | 1 : n << 1;             // sign in the lowest bit
  let out = '';
  do {
    let digit = v & 31; v >>>= 5;
    if (v > 0) digit |= 32;                               // continuation bit
    out += B64[digit];
  } while (v > 0);
  return out;
}

code

// test/transpiler.test.ts: the oracle is JavaScript itself
function evalBoth(src: string, expr: string) {
  const { code } = transpile(src);
  assert.ok(!ES6_SYNTAX.test(stripStrings(code)), `modern syntax left in:\n${code}`);
  const native = new Function(src + `\nreturn (${expr});`)();
  const es5 = new Function(code + `\nreturn (${expr});`)();
  assert.deepEqual(es5, native);                         // same behaviour, older dialect
}

Run it. In repos/transpiler: npm test, then npm run demo. This writes examples/app.es5.js and its map. Serve the folder, open examples/index.html with DevTools, and set a breakpoint in app.js: the browser runs the ES5 file and pauses in the original (the DevTools course part 3 covers source maps from the debugger's side).

exercises

1. Port the arrow rewrite to an AST from acorn and print with a code generator. 2. let in for loops with per-iteration bindings (wrap the body in an IIFE). 3. Shorthand properties and spread in calls. 4. Names in the map (the fifth VLQ field), so stack traces show original function names.