# Pre-Registered Test Strategy — Immunoglobulin Locus Explorer Bench

**Status: DRAFT for freezing.** To be **reviewed and signed off by a non-competitor**
(the human, or a neutral fourth model) **before any of the three apps is judged.** Once
frozen, no check, fixture, expected value, tolerance, or scoring weight below may change
after the first app's output has been observed. Any unavoidable mid-run correction MUST be
appended, dated, and justified in [§12 Mid-Run Fix Log](#12-mid-run-fix-log) — never edited
silently into the body.

This is a **plan for the harness (run by the Gemini 3.5 Flash / Antigravity agent) to
implement and run.** It does not itself run the apps.

**Subjects under test (SUT):** three independent implementations at
`./sites/claude/`, `./sites/agy/`, `./sites/codex/` — each a static web app whose entry
point is `index.html`, opened via `file://` **and** served from a nested subfolder
`/<repo>/sites/<tool>/` (subfolder-hosting is itself a test, §6E).

The strategy binds to the `./sites/<tool>/index.html` contract, not to any tool's internal
file names.

---

## 0. The shared brief (what all three were asked to build)

Identical one-liner to all three, biologist end-user:

> A standalone page giving a pleasant overview of the **immunoglobulin locus in mouse and
> human** (click to toggle), that **zooms all the way down to codons and nucleotides**;
> **homologous V regions coloured alike**; **easy zoom in/out so the user doesn't get lost**;
> and a **guided-but-escapable mode** to pick a **V/D/J (heavy)** or **V/J (light)** and see
> **how it comes together**, allowing **N- and P-nucleotide addition**.

Everything in that sentence is **core** and testable for all three; nothing here is tailored
to a capability only one tool happens to have. Where a tool omits a core element, it is scored
as a **feature-coverage gap** (points off coverage), **never** excluded as N/A — excluding it
would reward minimalism (the d002 pathology).

---

## 1. Conflict of interest & fairness protocol

- **COI disclosed (loudly):** **Claude built one of the three subjects** (`./sites/claude/`)
  **and drafted this strategy.** This is a self-interested grader. Per the build-off rules a
  perfect or top score for Claude here is a flag to **scrutinize, not publish** — attach the
  COI caveat and prefer a neutral re-review.
- **Mitigations, all pre-registered:**
  1. **Non-competitor freeze.** This document is signed off by the human or a neutral model
     **before** any subject is judged. The three contestants never approve it.
  2. **Byte-identical grading.** One harness, one adapter interface, one reference oracle, one
     tolerance set, applied to all three. No per-tool code in scoring. The only per-tool code
     permitted is the thin DOM **adapter** (§9) — pure I/O plumbing, no biology, no tolerance,
     no scoring.
  3. **Blind expected values.** Every expected sequence, translation, and CDR3 anchor in §6
     comes from the reference panel (§3), computed from IMGT-derived germline + the standard
     genetic code — **not** read out of any subject app.
  4. **Policy-vs-bug firewall (§10).** Divergence on genuinely underspecified questions
     (e.g. how many V genes to draw, exact intergenic spacing, which isotypes to label) is a
     **policy call**, not a correctness failure — so the grader cannot quietly penalise a rival
     for a defensible design choice.
  5. **Public + reproducible.** Harness, adapters, oracle panel, fixtures, and this document
     ship in the repo; a third party can re-run and diff.

---

## 2. Environment probe (run FIRST — record the result)

Before judging anything, the harness records its own environment:

1. Is **Python + a germline/IMGT reference library** (e.g. `receptor-utils`, `python-igblast`,
   local IMGT/GENE-DB dump) available? **Expected: NO** — unlike molecular weight, there is no
   ubiquitous pip oracle for Ig germline/V(D)J.
2. Because there is no authoritative offline oracle, the harness uses the **frozen reference
   panel in §3**, re-implemented from published IMGT germline sequences and the standard codon
   table, and labels every derived expected value **"reference, not authoritative."**
3. Record: Node version, headless-browser engine + version, OS, and whether a network egress
   monitor is active (needed for §6E "no undeclared requests").

---

## 3. Reference oracle — definition

A small, frozen **reference panel** committed under `./test/reference/`:

- **Germline sequences** (5′→3′, coding): a curated set of **IMGT/GENE-DB-derived** V, D, and
  J segments for human & mouse **IGH / IGK / IGL** — at minimum the textbook set:
  human `IGHV3-23`, `IGHD3-10`, `IGHJ4`, `IGHJ6`, `IGKV1-39`, `IGKJ1`, `IGLV1-40`, `IGLJ2`;
  mouse `IGHJ2`, `IGKJ1`, and representative V/D. Each row: name, locus, type, sequence, source
  accession.
- **The standard genetic code** (DNA codon → amino acid) as the single translation authority.
- **V(D)J landmark rules** (used to grade the recombination feature):
  - CDR3 opens at the **conserved 2nd-Cys** in V (codon `TGT`/`TGC`) and closes at the
    **conserved J anchor** — `Trp` (`TGG`) for heavy, `Phe` (`TTT`/`TTC`) for light — each
    followed by Gly.
  - **12/23 rule:** a 12-spacer RSS recombines only with a 23-spacer RSS; hence IGH cannot join
    V→J directly (D is obligatory). Expected spacer partners are tabulated per locus.
  - **P-nucleotides** are the reverse-complement palindrome of the coding end; **N-nucleotides**
    are non-templated (TdT). A join is **productive** iff the assembled V→J reading frame is a
    multiple of 3 **and** carries no premature stop through the J anchor.

### 3a. Oracle self-test (must PASS before any subject is judged)
Translate each panel V in frame 0 and assert the framework tail ends `…YYC` (heavy/κ) /
`…YFC`/`…YYC` (λ) at the conserved Cys; assert `IGHJ4` frame-2 reads `…WGQGT…`; assert the
documented 12/23 partners are internally consistent. Log all self-test results.

---

## 4. Measurement principle — separate correctness from presentation

- **Sequence / translation / CDR3 checks compare parsed tokens, not pixels.** Read the tool's
  reported nucleotides, amino acids, or CDR3 via its DOM adapter and compare to the panel
  **exactly** (sequence) or **residue-for-residue** (protein).
- **Formatting is usability, never a correctness failure.** Whether a tool writes `CAR…` vs
  `C-A-R`, shows lowercase intron filler, or spaces codons — that is §6-E, not §6-A. (The d002
  lesson: Codex's rounding wrongly tanked its correctness until display was firewalled from
  math.)
- **A missing coordinate system is not a wrong one.** These tools use *schematic* loci with
  compressed spacing; do **not** grade genomic coordinates against a real assembly. Grade
  **relative order** (V→D→J→C), **membership**, and **sequence-level** content instead.

---

## 5. Harness architecture — black-box via per-tool DOM adapters

- Each subject gets a thin `adapter.<tool>.mjs` exposing a common interface, e.g.
  `setSpecies(s)`, `setLocus(l)`, `zoomToNucleotide(segId)`, `readVisibleSequence()`,
  `pickVDJ({v,d,j})`, `addN(region, seq)`, `addP(region, n)`, `readAssembly()`,
  `readCDR3()`, `readProductive()`.
- Adapters are **pure plumbing**: locate DOM nodes / drive the tool's own controls and read
  text back. **No biology, no tolerance, no scoring** lives in an adapter.
- If a tool lacks a capability the interface needs (e.g. no nucleotide zoom, no recombination
  UI), the adapter returns `UNSUPPORTED`; the check records a **coverage gap** (§0), not N/A.
- Driven headless (Playwright/Puppeteer). Each tool loaded **twice**: once via `file://`, once
  from a nested `/<repo>/sites/<tool>/` path (§6E).

---

## 6. The checks (≥20, plainly named, grouped by the eight biologist questions)

Getting-the-biology-right dominates. Weights in §7.

### A. Gets the answer right — *dominant weight*
- **A1 · Germline sequence fidelity.** For every segment the tool exposes at nucleotide level,
  the shown sequence matches the reference panel (for segments the tool claims are "real/IMGT")
  or is clearly labelled synthetic/representative (graded under C, honesty).
- **A2 · Translation correctness.** Amino-acid translation of any displayed coding segment
  matches the standard genetic code, in the tool's stated frame.
- **A3 · Conserved CDR3 anchors.** In a V(D)J/ V-J assembly, the CDR3 the tool reports starts
  at the conserved Cys and ends at the conserved Trp (heavy) / Phe (light).
- **A4 · Reading-frame / productivity logic.** Adding 1 then 2 then 3 N-nucleotides flips the
  join out-of / out-of / back-in frame; the tool's "productive?" verdict tracks (multiple-of-3
  **and** no premature stop). Tools without a productivity verdict take a coverage gap on A4.
- **A5 · P-nucleotides are palindromic.** P-nt the tool adds equal the reverse-complement of
  the adjacent coding end.
- **A6 · 12/23 rule represented.** Heavy chain requires a D between V and J (no direct V→J);
  the tool does not offer a biologically impossible direct heavy V→J productive join, or
  explains the constraint.
- **A7 · Homologous V colouring is consistent.** V segments of the same family share a colour,
  and the *same family* is coloured comparably across mouse↔human (the explicit brief ask).
- **A8 · Locus content & order.** Human & mouse IGH/IGK/IGL each show V→(D)→J→C in the correct
  order, D only in heavy, and the correct constant-region **isotypes** present (IgM/IgD/IgG…/
  IgA/IgE for heavy; single Cκ; multiple Cλ).

### B. Survives messy data / messy interaction
- **B1 · Deep zoom doesn't break.** Zooming from whole-locus to a single nucleotide and back
  never NaNs, blanks, or throws (console clean).
- **B2 · Species/locus toggle mid-zoom.** Switching species or locus while zoomed in doesn't
  desync the view or crash.
- **B3 · Empty / extreme recombination.** Requesting an assembly with missing picks, or huge N
  strings / non-ACGT N input, degrades gracefully (no crash, no silent garbage).
- **B4 · Rapid input.** Fast repeated zoom/pan/toggle stays responsive; no unbounded listeners
  or leaks.

### C. Honest about what it did
- **C1 · Real vs schematic disclosed.** The tool states which sequences are reference germline
  vs synthetic/representative, and that spacing/coordinates are schematic — no implied genome
  accuracy it doesn't have.
- **C2 · Non-productive is labelled, not hidden.** Out-of-frame or stop-containing joins are
  shown as non-productive, not silently "fixed."
- **C3 · No confident-wrong.** No mislabelled isotype, no V family shown under the wrong locus.

### D. Handled the vague ask well *(judged from app + its README/stats)*
- **D1 · Sensible documented assumptions** (how many V genes drawn, why compressed spacing).
- **D2 · Guided mode is guiding but escapable** — a beginner is led to pick V/(D)/J, yet can
  work free-form and exit the guide (the explicit "not heavy-handed" ask).
- **D3 · Graceful degradation over gold-plating** — extras don't get in the way of the core.

### E. Usable · fast · safe · holds up
- **E1 · Obvious on load** (clear entry, labels, legend for the colours).
- **E2 · Keyboard + screen-reader** operable controls; adequate contrast; results announced.
- **E3 · Performance.** Median of 5: initial render and a deep-zoom interaction stay responsive
  (no multi-second lock) on the largest locus (mouse IGH). Compared only across the three.
- **E4 · Subfolder-hosting.** Served from `/<repo>/sites/<tool>/`, **zero missing assets**
  (grep the built pages for root-absolute `src="/`, `href="/`, `url(/`) and **zero undeclared
  network requests** (offline egress monitor is silent, or every request is disclosed).
- **E5 · Runs from `file://`** with no server, account, install, or key.
- **E6 · Self-contained & clean** — small enough to share; no obvious security foot-gun
  (e.g. `innerHTML` of user-pasted text without escaping → the code-injection gate below).

### Adversarial gate (must pass, not weighted)
- **X1 · Code-injection.** Paste `<img src=x onerror=alert(1)>` / `"><script>…` anywhere the
  tool accepts text (sequence box, N-nucleotide field). It must be inert (escaped/ignored),
  never executed.

---

## 7. Scoring model (frozen weights)

Per the eight biologist questions; getting-the-answer-right dominates. Each check scored
0 / partial / 1 against its written pass condition; a coverage gap scores 0 on that check
(counted in the denominator).

| Question | Checks | Weight |
|---|---|---|
| 1 · Right answer | A1–A8 | **32%** |
| 2 · Survives messy data | B1–B4 | 17% |
| 3 · Honest | C1–C3 | 14% |
| 4 · Handled vague ask | D1–D3 | 10% |
| 5 · Usable | E1–E2 | 10% |
| 6 · Fast enough | E3 | 7% |
| 7 · Safe & runnable | E4–E5 + X1 | 6% |
| 8 · Holds up | E6 | 4% |

`X1` is a **gate**: failing it caps the tool's "Safe & runnable" block at 0 and is flagged
prominently regardless of total.

---

## 8. Fixture corpus (defined HERE — frozen)

- **Canonical:** the panel segments of §3 (human IGHV3-23 / IGHD3-10 / IGHJ4; IGKV1-39 / IGKJ1;
  IGLV1-40 / IGLJ2; mouse IGHJ2 / IGKJ1).
- **Assemblies (expected CDR3 written out):** e.g. human heavy `IGHV3-23 + IGHD3-10 + IGHJ4`
  with 0 N → expect a `CAR…YFDYW`-shaped, in-frame CDR3; +1 N → out of frame; light
  `IGKV1-39 + IGKJ1` with the N needed to reach frame → `C…F`-shaped. (Exact expected strings
  filled in at freeze time from the panel.)
- **Adversarial:** the X1 injection payloads; non-ACGT N input; lower-case/space-laden paste.
- **Performance:** mouse IGH (largest V array) — initial render + one deep-zoom, median of 5.

---

## 9. Verify-before-publish — classify every negative finding

Before any finding is published, classify it:

- **Real bug** — reproducibly wrong sequence/translation/CDR3, a crash, or an executed
  injection. Keep; show the repro.
- **Debatable policy call** — a defensible design divergence (V-gene count, spacing, which
  isotypes to draw, guided-mode ergonomics). Re-label "divergent, defensible."
- **Harness artifact** — the check or adapter is wrong. **If all three fail the same check,
  suspect the check first.**
- **Precision ≠ wrong biology** — a display/format limit is usability, never correctness (§4).
- **Coverage gap ≠ N/A** — a missing core element counts against coverage; never dropped from
  the denominator.
- **Scrutinise a top Claude score** — if `./sites/claude/` lands at/near the top, attach the
  COI caveat and prefer a neutral re-review before publishing.
- **Frozen-artifact check** — confirm only the harness changed between runs (file mtimes); a
  tool edited after results were seen is a fairness violation.

No security claim (X1 or E4) without an end-to-end reproduction.

---

## 10. Outputs (emitted by the run, not by this plan)

Written into `./test/` (mirroring the published `stats/` + `testing/` layout):

- `results.md` — per-tool narrative, findings classified per §9.
- `correctness-matrix.md` — every check A1…X1 × {claude, agy, codex}, pass/partial/gap, written
  out in plain language (no cryptic codes).
- `results.csv` — machine-readable check-by-check.
- `verification.md` — the §9 classification log + environment probe + oracle self-test.
- Legends/labels use **model names** (Claude Opus 4.8 · Gemini 3.5 Flash · GPT-5.5); folder ids
  (`claude`/`agy`/`codex`) stay internal.

---

## 11. Handoff note to the harness author (Gemini 3.5 Flash / Antigravity)

Implement §5 adapters against each tool's real DOM; wire the §3 panel + §8 fixtures; run §6;
emit §10. Do **not** invent expected values — take them from the frozen panel. Where a tool
can't do something, record `UNSUPPORTED` → coverage gap, and move on. Keep the adapter free of
any biology or scoring so §1.2 (byte-identical grading) holds.

---

## 12. Mid-Run Fix Log

*(empty at freeze — every post-freeze change is appended here, dated and justified, never
edited silently into the body above.)*