Post-mortem: we spent nearly eight months testing TypeScript's compiler

TJS converts TypeScript in two steps: fromTS turns TypeScript into TJS, and tjs turns TJS into JavaScript. This project checks that pipeline with several test suites it calls lanes. Three of them were cited as evidence that the converter works:

• the compat lane, which converts real TypeScript projects (zod, effect, ts-pattern and others) and runs each project's own test suite on the result;
• the Bootstrap Canary (src/use-cases/bootstrap.test.ts, whose last section is titled "True self-hosting"), which converts the toolchain's own TypeScript source and runs it;
• the dogfood ratchet, which converts this repository's whole source tree.

For most of the project's life, the compat lane, described as "the most honest evidence the converter works that this repo has", was in the main evidence that tsc works. So was the Bootstrap Canary. Neither was lying; both were pointed at the wrong program.

Nothing was broken in a way anyone could see. That is the point of writing it down.

Why a TypeScript evaluator should read this. The compatibility figures this project now quotes for zod, effect, ts-pattern and the rest measure TJS's own converter, and this page is the record of how we know that. Until 2026-08-29 the JavaScript in those runs came from TypeScript's compiler, so the numbers of that time said little about TJS. If you are deciding whether to trust a converter with your codebase, how its evidence failed and was repaired tells you more than the evidence alone.

§1–§7 are the analysis. §8–§10 are a dated log of the repair and what was measured afterwards; the current figures are in §10.

1. What happened

fromTS had two branches:

if (emitTJS) { tjsFunctions.push(transformFunctionToTJS(...)) }   // TS -> TJS
else         { metadata[funcName] = extractFunctionMetadata(...) } // TS -> JS, via tsc

The else side (call it the tsc branch) produced JavaScript with ts.transpileModule — TypeScript's own emitter, sitting beside ours — and stapled fn.__tjs = {…} onto the result.

Two JavaScript emitters is a maintenance smell. That was the small half. The large half is that two of the three lanes we cited as evidence called the tsc branch: the compat lane and the Bootstrap Canary. Only the dogfood ratchet ran TS → TJS → JS through our own code.

2. Timeline

Without --full, each compat script asked fromTS for JavaScript directly, which meant the tsc branch. With it, the script composed fromTS → tjs. The flag defaulted off from the compat lane's first commit, and nothing ever passed it: when compat-all.ts arrived it spawned every script with no arguments, and three of the six scripts never had the flag at all.

3. Why it survived

Because the tsc branch was load-bearing. It was the only side that collected type metadata — the TJS side returned no types at all. So anything that wanted types and a transpile was forced through TypeScript's emitter. Deleting it was impossible without first noticing that the two branches were not two renderings of one analysis but two separate programs that never both ran.

That is the generalizable shape: a redundant path that has one capability the good path lacks is not redundant. It is a dependency.

And a comment said otherwise. The Bootstrap Canary read:

// Transpile with TJS
const result = fromTS(source)

That fromTS call, with no options, took the tsc branch, so the line under the comment ran ts.transpileModule.

4. What it cost

Forcing every lane onto our own emitter turned 52 tests red. Those 52 collapsed to four defects, all previously invisible. Two more turned up a day later, once zod could be measured at all; §8.2 lists all six.

In TJS a colon value is an example, not a type: a: 0 means "an integer, for example 0", and 0.0 is the example for a float. A return written :! is a type-only example: it is not run as a test. That is the syntax the converter emits, and the syntax these defects are about:

• TJS could not parse an annotated generator (function* f():! 0.0) — the function matcher did not admit the *, and once it did, the * was dropped while rebuilding the header, so every yield became a reserved word.
• TJS could not parse a destructured parameter with an inline type (function f({ a, b }: { a: 0, b: 0 })) — it matched as one pattern, so the colon-shorthand rewrite ran inside the type.
• An optional parameter converted to a required one. excited?: boolean became excited: boolean | undefined, which preserves the type but is required — and the emitted __tjs said required: true while fromTS's own metadata said required: false.
• Interface members of unresolvable type emitted any. A default for an optional object parameter must be made of literal values (TJS builds the merge of defaults from the declaration, so a default it cannot copy as data is a compile error), and any is not one. Such a parameter was therefore rejected at graduation, the step that turns a converted file into native TJS by removing its JS-semantics annotation.

The converter was producing TJS that TJS could not read, and no lane could see it, because the lane that exercised the converter's output never fed it to our parser.

Re-running the compat lane on the real path found 15 files that failed to transpile across five real codebases (zod could not be measured yet), in two classes: declaration-merging name collisions, and TS overloads our polymorphic dispatch reads as ambiguous. The old numbers — ts-pattern 453/453 upstream tests passed, kysely 303/303 files transpiled, effect 363/363 files transpiled — measured tsc. (§8.3 is a different and wider measurement: every file of six projects, zod included.)

5. Why the gap stayed small

Despite nearly eight months unseen, only six defects (§8.2) had piled up between what the lanes measured and what was true. Two things kept the gap small:

1. Only half the pipeline had a bypass. TS → TJS was never shortcut — the CLI convert and the playground always used it. Only TJS → JS had the shortcut.
2. The dogfood ratchet applied real pressure for its four weeks of life, driving our whole source through TS → TJS → JS. A ratchet here is a minimum pass count that may only go up; its floor, the number of this repo's source files that must convert, compile and graduate, rose from 95 to 106.

But the sharp version, in files containing the shape (as counted on 2026-08-30; the generator figure for our own source was an estimate, not an exact count):

                                    our source   compat corpora
destructured param w/ inline type        0             6
annotated generators                    ~2             1

The honest lane had a narrow corpus; the broad corpus was on the dishonest lane. The dogfood ratchet could never have found the destructuring bug — our own code does not contain the shape. radash and effect do. Coverage and honesty were inversely distributed, which is worse than either gap alone and invisible from inside either one.

6. Lessons

Ordered by how transferable they are.

1. Name the program under test. A lane that runs someone else's implementation measures theirs. "Does the converter work?" and "does tsc work?" have different answers and looked identical for nearly eight months.
2. A redundant path with a unique capability is a dependency. The tsc branch survived because it was the only source of type metadata. Look for what the "obsolete" path can do that the good one cannot; that is what is keeping it alive.
3. The default is the only configuration that matters. --full existed, worked, and was never once passed. An option that must be remembered will be forgotten; make the honest path the only path.
4. A comment is not a control. // Transpile with TJS sat directly above a call to tsc and was read by everyone who touched that file.
5. Green that cannot go red is not a signal. Even after the fix, compat-all printed 5 passed, 0 failed while superstruct ran zero tests and ts-pattern failed on source that never transpiled. The scripts exit 0 regardless. Before quoting a number, check what would have to happen for it to fall.
6. Ask what the honest lane's corpus contains. Full marks on a corpus that lacks the shape is not evidence about the shape.
7. When you cannot tell a converter defect from a language defect, you will blame the language. During the fix, the agent doing it (an AI coding assistant) claimed a language bug three times — "optional object parameters are not writable in TJS". The spec, docs/dictionary-defaults.md, had settled it a month earlier: whether a parameter is required belongs to the parameter, not its members, so (opts: {…}) is required and (opts = {…}) is optional with every member defaulted. The language was correct every time; every failure was the converter emitting something the spec already forbade. Read the spec before diagnosing, not after someone points at it.

7. What now enforces this

• src/no-ts-emitter.test.ts — fromTS must refuse to emit JavaScript (with a control proving the composed path works), and only from-ts.ts may call ts.transpileModule at all. The remaining call sites, where tsc still strips types from function bodies, are a ratchet that may only go down: the count may not exceed a ceiling (currently 1), and a promote-check fails when the count drops below the ceiling, so the ceiling has to be lowered with it. That dependency is real, and shrinking it to zero means writing our own TypeScript statement stripper.
• One path. fromTS does TS → TJS; tjs does TJS → JS; callers compose them, so the route is visible at the call site. emitTJS: false throws and names the replacement.
• All six compat scripts run the real path, with no flag to get it wrong.
• The Bootstrap Canary genuinely self-hosts, and its comment is true.

What a caller writes now:

import { fromTS } from 'tjs-lang/lang/from-ts'
import { tjs } from 'tjs-lang/lang'

const ts = 'function add(a: number, b: number): number { return a + b }'
const tjsSource = fromTS(ts).code
// /* tjs <- input.ts */
// …
// function add(a: number, b: number):! number { … }
const js = tjs(tjsSource).code

fromTS(ts, { emitTJS: false })
// throws: fromTS no longer emits JavaScript — it emits TJS. Compose the two steps: …

The conversion changes less than it looks. TJS accepts bare type names such as number, so the parameters survive as written. The return becomes :! number, a type-only example, because a TypeScript return type is not a worked example to test. And the /* tjs <- input.ts */ header marks the file as converted, which means it keeps JavaScript semantics: no input validation unless you opt in (/* @tjs TjsStrict */) or graduate it.

8. Remediation

Recorded as it happened, because "we fixed it" is the part that rots first.

8.1 The pipeline had multiplied, not just drifted

The regression was not only wrong, it was more work to be more wrong. Three duplicated paths, each a place for the two halves to disagree:

The principle that keeps falling out: we do not convert TS to JS. We convert TS to TJS. Everything downstream of that is one path. The browser entry (browser-from-ts.ts) needed no change — it delegates to fromTS, so it inherited TS → TJS and can no longer emit JS even by accident. The TS playground (demo/src/ts-playground.ts) already composed correctly.

8.2 What the honest lanes then found

Each of these is the same shape — valid input the converter produces and the language could not read — and each was invisible while tsc did the emitting. The first four are the defects behind the 52 red tests in §4; the last two appeared on 2026-08-30, once zod could be measured.

The last two are worth separating: they are not new rules, they are existing rules that were never applied to the neighbouring case. Look for the sibling site.

8.3 The corpus, measured honestly

The first honest measurement, from 2026-08-30: files that convert and parse, across six real TypeScript projects. This probe did not cover whole trees. Four projects show exactly 150 files and the total is 634, against 1973 files in the full scan of §10, so it was sampled (most likely capped at 150 files per project; the probe itself was not committed) and 89% is a figure over that sample.

That leaves 71 failing files. The causes recorded at the time account for 70 of them: 57 parse, 10 ambiguous-overload, 3 already-declared. The record does not give a cause for the 71st. effect is the laggard, dominated by class X extends Schema.Class<X>("X")({…}) — a class extending a call expression with type arguments.

Do not compare these to the old numbers. ts-pattern 453/453 counted upstream tests passed and kysely 303/303 counted files transpiled, and both measured tsc. This table counts files that our converter turns into JavaScript that parses. 89% is the first figure this project has published about its own converter. §10 records where it went next.

8.4 A probe that failed open, again

The first version of the corpus probe printed 0/0 files (100%) for every target — its find excluded *test*, which matched the .compat-tests directory itself. It read as a clean pass. Fixed by making an empty file list print NO FILES — probe broken, not a pass. That makes three measurements on this page that failed silently in the passing direction, after --full defaulting off (§2) and compat-all reporting 5 passed while superstruct ran zero tests (lesson 5). Assume it of every new probe until shown otherwise.

9. Still open

Updated 2026-11-06. Each item as it was left in §8, and where it stands now.

• The lane verdict (lesson 5). Closed. scripts/compat-all.ts runs every compat script, prints a table, and exits non-zero if any target fails. An unmet prerequisite (zod needs pnpm) is reported as a skip, not a failure, because it says nothing about the converter. Each script also fails when its own transpile step fails, and the vitest-based scripts fail on a suite that did not load, not only on failed tests.
• The two compat failure classes, and superstruct src/utils.ts:188 (a one-off Unexpected token parse failure in the compat lane, never diagnosed at the time). Closed. The per-file scan reached 1973/1973 on 2026-09-02 (§10).
• The ts.transpileModule body-stripper ratchet, down to zero. Open. One call site remains, inside the TS → TJS step, where tsc strips type annotations from function bodies. src/no-ts-emitter.test.ts holds it at a ceiling of 1. Reaching zero needs our own TypeScript statement stripper.
• Two model-selection implementations (§8.1). Partly open. The demo's vision probe now uses the batteries' shared checkVision; its chat-model resolution in demo/src/capabilities.ts is still its own.

10. Since then

Recorded so the figures in §8.3 are not read as current. Both lanes below run before every release, as part of the suite bun run release:ready runs locally and attests; they do not run on every commit or in CI. The figures are the last full runs as of their dates.

Every file converts and parses. bun run test:compat-scan (scripts/compat-scan.ts) runs every non-test, non-declaration .ts source file in the six cloned projects (skipping tests/, dist/ and node_modules/) through TS → TJS → JS and checks that the result parses. It is a ratchet in both directions: a file that newly fails is a regression, and a file on its known-failure list that starts passing must be removed from the list. On 2026-09-02 it reached 1973/1973 files across the whole corpus and the known-failure list became empty. Files over 400KB are skipped and named in the output, because the preprocessor is quadratic in file size and hangs on them rather than failing. A skip that reads as a pass is the failure this page is about, so the scan says so.

The projects' own test suites pass on the converted code. bun run test:compat clones each project, converts it with fromTS and tjs, and runs that project's test suite against the result. The last full run, on 2026-09-25:

Two of these deserve a note. zod's first "pass" was not one: 105 of its suites failed to load, and a suite that cannot load has no failed tests, so the harness counted it as green. That is lesson 5 again, and it is why every vitest harness now fails on a failed suite. And ts-pattern 453/453 is the same number §4 dismissed. Then it measured tsc; now it measures our converter.