Continuity Bench
How well do models detect continuity errors in novel-length fiction, and what does each detection cost?
The question
Can a cheap non-generative classifier, escalating to an LLM only when unsure, match full-context LLM accuracy on continuity errors for a fraction of the cost?
Not measured for Jev itself: no Jev run has been recorded, so no number on this page is Jev’s.
What is measured: with a stand-in in Jev’s slot, local_oracle_llama3.1:8b_contradiction clears zero — the only interval on this board that excludes it. That run is handed a hand-built state, so it is the architecture’s ceiling rather than a runnable pipeline. Whether Jev clears it is a separate question, and an unrun one.
Leaderboard
294 items from 31 novels, corpus continuity_v0.jsonl
How each run behaves
Recall against false-positive rate. The diagonal is guessing: distance above it is J, position along it is only how readily a run flags anything. Top-left is the good corner.
F1, for comparison
Dashed line: flagging every passage scores 0.667 while discriminating nothing. This is why F1 is not the headline.
No run has cost API money yet, so there is no cost axis to draw. The cost-vs-quality view appears once a paid or hosted run lands.
| predictor | J | J 95% CI | recall | FPR | AUC | F1 | precision | localization | $/passage | items |
|---|---|---|---|---|---|---|---|---|---|---|
| laya_english | +0.019 | -0.139 to +0.176 ‡ | 0.325 | 0.521 | 0.236 | 0.217 | 0.000 | $0.00000 | 212 | |
| always_clean | +0.000 | -0.025 to +0.025 • | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | $0.00000 | 294 | |
| always_error | +0.000 | -0.025 to +0.025 • | 0.667 | 0.500 | 1.000 | 1.000 | 0.020 | $0.00000 | 294 | |
| local_llama3.1:8b_high | +0.000 | -0.061 to +0.061 • | 0.662 | 0.500 | 0.981 | 0.981 | 0.365 | $0.00000 | 212 |
• the interval is tight and contains zero: that run is measured to have no discrimination. ‡ the interval is wide and contains zero: that run establishes nothing either way, usually too few items.
How this is measured
Continuity errors are injected into passages of public-domain novels at a known location, and every injected passage ships with the same passage unedited as its control, so the false-positive rate is measured on exactly the same text as the recall. Splits are assigned per novel, so no novel appears in both dev and test.
The headline metric is J = recall − false-positive rate, not F1. On a balanced paired corpus, flagging every passage scores precision 0.500, recall 1.000 and therefore F1 0.667 while discriminating nothing at all — a real model has scored exactly that here. J is 0 for any such constant strategy and 1 for a perfect one.
Every run carries a 95% Wilson interval. An interval that is tight and contains zero means the run is measured to have no discrimination; one that is wide and contains zero means the run establishes nothing. Those are marked differently because they mean opposite things.
Before any model was run, the corpus was attacked with zero-cost heuristics, including one that pools novel-wide name statistics the way someone who downloaded the dataset would. The best reached J +0.066, so the benchmark is not simply leaking its own injection method.
Benchmark integrity
The strongest zero-cost heuristic reaches J +0.020. A benchmark whose errors could be found by a regex would not be measuring reasoning.
The diagnostics exploit the release format rather than the text: pair_diff_targeted reaches J +0.408 by diffing the two halves of a matched pair. That is why the published test split ships one half of each pair and holds its labels back.
The ceilings are fed a hand-built story state instead of one the pipeline extracted: local_oracle_llama3.1:8b_contradiction reaches J +0.264 that way. They bound what the architecture could do given perfect state, which is not what a deployed system has, so they are reported here rather than ranked.
| predictor | J | J 95% CI | recall | FPR | AUC | F1 | precision | localization | $/passage | items |
|---|---|---|---|---|---|---|---|---|---|---|
| pair_diff_targeted | +0.408 | +0.307 to +0.489 | 0.580 | 1.000 | 0.408 | 0.000 | 1.000 | $0.00000 | 294 | |
| local_oracle_llama3.1:8b_contradiction | +0.264 | +0.075 to +0.435 | 0.645 | 0.623 | 0.670 | 0.406 | 0.437 | $0.00000 | 212 | |
| pair_leak | +0.150 | -0.008 to +0.300 ‡ | 0.618 | 0.561 | 0.687 | 0.537 | 1.000 | $0.00000 | 294 | |
| laya_oracle_english_contradiction | +0.066 | -0.116 to +0.243 ‡ | 0.589 | 0.526 | 0.670 | 0.604 | 0.169 | $0.00000 | 212 | |
| trait_disagreement | +0.020 | -0.018 to +0.058 • | 0.040 | 1.000 | 0.020 | 0.000 | 1.000 | $0.00000 | 294 | |
| corpus_prior | +0.007 | -0.117 to +0.130 ‡ | 0.270 | 0.509 | 0.184 | 0.177 | 0.296 | $0.00000 | 294 | |
| narration_person | +0.000 | -0.160 to +0.160 ‡ | 0.495 | 0.500 | 0.490 | 0.490 | 0.111 | $0.00000 | 294 | |
| rarest_name | +0.000 | -0.025 to +0.025 • | 0.667 | 0.500 | 1.000 | 1.000 | 0.211 | $0.00000 | 294 | |
| singleton_name | +0.000 | -0.093 to +0.093 • | 0.646 | 0.500 | 0.912 | 0.912 | 0.149 | $0.00000 | 294 |
What this does not show
- Injected errors are not natural errors. This measures detection of programmatically injected contradictions.
- The novels are famous, so they are in every frontier model's training data. Per-novel recall is published in each run's JSON so memorisation shows up as variance.
- One error type dominates. Regex injection over real prose supports one high-volume type plus a small tail; balanced volume needs an LLM injector.
- Local models report $0.00 because they cost no API money. They cost wall clock and hardware instead, which is why median latency sits beside the cost.