Testing a 68000 Emulator Against SingleStepTests: 317,500 Vectors

Written by Claude (Opus 5.5) and checked by Claude (Fable 5.2), from the repo, its notes and the session transcripts. Earlier Claude sessions wrote the brief and the CPU core; dested set the project going and made the calls. The first person below is dested's voice.

An agent wrote me a Motorola 68000 interpreter in TypeScript and ran it against all 317,500 vectors in the SingleStepTests suite. The first full run got 309,067 right. Of the 25 files with misses, 23 failed for the same reason: what the chip leaves behind when an address error stops an instruction halfway.

The core is part of gens-dev, a private project that simulates a 1993 Genesis dev kit in the browser. The brief's bar: pass every vector, cycle counts and bus order included, and run at 40 M cycles a second or better. The Genesis needs 7.67.

#Every vector checks the bus, not just the result

SingleStepTests/m68000 is 127 files of 2,500 vectors each, in the format Tom Harte's tests use, generated from MAME's microcoded 68000. A vector is one instruction:

Registers start random, so odd addresses are common. In 55,606 of the 317,500 vectors, about one in six, the instruction ends in an address error. That means the suite grades the exception stack frame, the status word inside it, and every register an instruction had already touched before it faulted.

#The agent read the tests before writing the core

It cloned the suite, then spent its first stretch writing small scripts against the data: dump a vector, histogram the address-error status words, list the bus pattern of every addressing mode, tabulate DIVU timings. Only then did it write the core: 2,171 lines. A full run of all 317,500 vectors takes 4.4 seconds under Bun, and the core benchmarks at 188 to 218 M cycles a second.

#23 of the 25 failing files were address errors

Every one of those misses is on a faulting path. They change what's on the stack and in the registers after a fault, not what a working program computes. The other two were LINK A7, a plain semantics bug, and DIVU, which got every quotient right and 507 cycle counts wrong.

#The five patches, in order

Then a full run with bus order compared too: 317,500 of 317,500, no exclusions. The suite's README flags TAS and TRAPV as unverified, and both pass.

#The one bug that was JavaScript, not the chip

DIVU's cycle count comes from replaying the chip's shift-and-subtract loop. In C the subtraction wraps as unsigned 32-bit. In JavaScript it went negative, and every later comparison in the loop ran against a negative number:

// before: the bit shifted out of the top is lost
if (carry) dividend -= hdivisor
// after
if (carry) dividend = dividend + 0x100000000 - hdivisor

Any unsigned 32-bit arithmetic carried over from C has this trap in JavaScript.

#What a per-instruction suite can't see

A vector is one instruction from a set state, with no interrupt line. The agent wrote 14 tests of its own for what that leaves out: autovectored interrupts (level 4 through vector 28 in 44 cycles), the interrupt mask, level 7 as an edge-triggered NMI, STOP, trace, reset, the double-fault halt, breakpoints.

Two things are Genesis-specific. The console's bus drops the write half of TAS, so the core takes tasWritesBack: false there and true for the suite. And move.l #$40000000,($C00004).l must write the high word first or the VDP gets a garbage command; the vectors' bus order covers that, and a test pins it.

The next day the same agent wrote a 132-page 68000 reference manual for the dev kit, checked against the core by 225 tests. Its commit lists two things the core doesn't emulate: bus error (vector 2) and spurious interrupt (vector 24).

What I can't report is a CPU bug that only a game found. Sonic 1 booted headless to its title screen on this core, and the VDP rendered Sonic 1, 2 and 3 & Knuckles on it the same day. Git shows one later edit to the 68000 code: a five-line PC getter for the debugger link. The Z80 went the same way: 1,604,000 of 1,604,000 vectors.

#Wiring the suite into your own core

Clone the repo, port decode.py (the files are .json.bin), and run each vector like this:

for (const file of readdirSync(dir).filter((f) => f.endsWith('.json.bin'))) {
  for (const v of decodeVectors(new Uint8Array(await Bun.file(join(dir, file)).arrayBuffer()))) {
    bus.load(v.initial.ram)                    // clear only what the last vector touched
    cpu.setState({ ...v.initial, pc: v.initial.pc - 4 })
    bus.log = []
    const cycles = cpu.step()                  // one instruction, or one exception entry
    compareState(cpu.getState(), v.final, bus) // registers, SR, PC, prefetch, RAM, stray writes
    if (cycles !== v.length) cycleMisses++
    compareBus(bus.log, v.transactions)
  }
}

The traps:

• PC is MAME's address unit: the instruction's own address plus 4. Subtract 4 going in, add it back when you compare.
• Seed the prefetch queue from the vector, and compare it after.
• Compare RAM as 16-bit words, and flag any write to an address the final state doesn't list.
• Leave out n, re and we when comparing the bus. Idle cycles aren't accesses, and address-error cycles never reach memory. Split byte cycles by UDS/LDS.
• Count cycles separately from state. DIVU passed on state and failed only on cycles.
• Commit a smoke subset. The first 25 vectors of every file, gzipped, came to 770 KB and run in the normal test command with no network.

Then print one row per file: passed, failed, cycle misses, bus misses. That table is the whole debugging loop.