The study / Can a free local model replace Jev in Jevtown?

Can a free local model replace Jev in Jevtown?

Rendered from layatown/docs/EVALUATION.md in the project repository, 21 September 2026, word for word. Laya is an open model by Convai Innovations (Apache-2.0); Jev is TypeSafe's model. Not affiliated with either. Cited result files are listed in the study.

Pre-registered evaluation plan for Layatown, written 21 September 2026, before any fine-tuned checkpoint is scored on the held-out production day. It fixes the systems, data, metrics, statistics and pass/fail thresholds so that results cannot be chosen after the fact. It follows the project's rule for docs/EVALUATION.md: the plan is committed before the runs, never edited to fit results, and changed only by dated amendments at the end of this file.

1. Summary

Jevtown's citizens are driven by TypeSafe's Jev (jev-1.13.0), a paid System One model. Layatown is Laya (convaiinnovations/laya, ModernBERT-large, 421M parameters, Apache-2.0, revision 1c5edc17), fine-tuned by distillation on Jev's own logged probabilities from Jevtown (training/, see its data card training/README.md), and served on one local RTX 3080 Ti with the same POST /v1/systemone wire shape (layatown/serve.py).

The question is whether Layatown can take Jev's place in the live city. We ask it at three levels, from narrow to broad, because agreement on single answers does not guarantee that a city run on those answers behaves the same:

• A. Judgment fidelity. Does Layatown give Jev's answers, question by question, on a day of the city it never saw?
• B. Human-labelled checks. On the small sets where a person wrote the right answer, how do both models do? These include tasks Layatown was never trained on (chat and username screening, reading event proposals).
• C. City-level behaviour. Run the simulator with Layatown on the same scenarios and seeds that past Jev runs used. Does the village live, die, steal, befriend and diverge the way it did under Jev? Plus serving cost and latency.

No part of this plan requires new Jev spend.

2. Research questions

• RQ1 (fidelity). How closely does Layatown reproduce Jev's probability distributions on the held-out production day, per task, relative to the train-mean floor and to untouched Laya?
• RQ2 (character). Does Layatown preserve the trait and job effects that make Jev's citizens individuals (the study's pre-registered contrasts), or does it match Jev on average while flattening them?
• RQ3 (correctness where it is known). Where human labels exist, is Layatown as right as Jev? For tasks Layatown was not trained on, is it safe to let it answer them in the live city?
• RQ4 (closed loop). Does a village driven by Layatown produce story metrics closer to Jev's village than the hand-written rules baseline does, with the same survival and character effects?
• RQ5 (economics). What does one decision cost in time and energy on the local GPU, next to Jev's measured cost?

3. Systems

Before layatown-v1 is scored on the test day, the SHA-256 of its model.safetensors and its rl_agent_config.json (temperatures, training arguments) are recorded as an amendment. That checkpoint is frozen: no change to weights, temperatures, slicing or prompt format after the test day is opened.

4. Data

• Test (level A): training/data/test_production, the city's first public day (live-2026-09-19T23-51-37, 1,000 citizens, 435 ticks): 238,779 decision questions, 5,128 character sheets (8 yes/no answers each), 168 juror votes. Never used for training, temperature fitting, checkpoint selection or hyperparameter choice.
• Validation: training/data/val (4 whole live city runs). Used for temperature fitting, monitoring, and choosing among variants. Val numbers are not evidence for the claim; they are reported only as context.
• Human-labelled sets (level B): training/data/eval/*.jsonl (445 rows, 5 files).
• City runs (level C): runs/eval6-hard-week-*, runs/eval6-siege-*, runs/eval7-strange-week-* (policies jev, rules, random, seeds 1 to 5, 240 ticks, world jevtown, world hash a635a002c79cd306, which equals the current worlds/jevtown hash on 21 September 2026). These are the runs after the list-position fix (amendment of 2026-09-18 in docs/EVALUATION.md); earlier eval- to eval5- runs addressed citizens by list position and are not used.

5. Level A: judgment fidelity on the production day

5.1 Unit and estimand

The unit is one question (one citizen's decision, one yes/no line of a sheet, one juror's vote). Every metric is the item-weighted mean over all questions of the task in test_production, with no subsampling (--every 1). Items that data.py cannot build (an option marker falls off the sequence) are counted and reported; if more than 0.1% of a task's items are dropped, the result for that task is reported as incomplete.

5.2 Metrics (exact definitions)

Let p be the model's distribution and q Jev's target distribution over the same options, in criteria order.

villager_decision (choice, 8 to 11 options)

• TVD = 0.5 * sum_k |p_k - q_k|. Primary.
• top-1 agreement = 1[argmax p == argmax q] (ties in q broken by criteria order, as numpy does). Primary.
• KL(q || p) in nats, with p clamped at 1e-9. Secondary.
• Expected calibration error of the top choice against Jev's top-choice probability (10 equal-width bins of max p, gap = |mean max p - mean q[argmax p]|, weighted by bin size). Secondary, reported without threshold.
• Per-action marginals: mean p_k against mean q_k for every action. Reported, no threshold.

character_sheet and juror_vote (noul)

• MAE = |p_yes - q_yes|. Primary.
• Brier-to-teacher = (p_yes - q_yes)^2. Secondary.
• side agreement = 1[(p_yes >= 0.5) == (q_yes >= 0.5)]. Secondary.
• For sheets, MAE is also reported for each of the 8 sheet questions separately.

5.3 Character effects on the test day (RQ2)

Using the states of the test-day decision items, compute for each pre-registered contrast the difference effect = mean p(action | in-group) - mean p(action | others), once with Jev's targets and once with each model's predictions, on exactly the same items. Group and item selection follow the definitions in bun report (src/cli/report.ts):

Jobs or traits absent from the city world are reported as not applicable, not as passes.

5.4 Floor

For each choice question, the floor predicts the mean train target over train questions with the same ordered option set; for nouls, the train mean for that task (0.39 for sheets, 0.49 for jurors). The data card reports TVD 0.517, top-1 0.278, sheet MAE 0.142 and juror MAE 0.112 on the test day. Because no committed script currently produces those numbers, the evaluator recomputes the floor on the identical item set; both the card's and the recomputed values are reported, and thresholds that reference the floor use the recomputed value.

5.5 Uncertainty: bootstrap over ticks

Questions within one tick share the village state, the moment's events and often the same citizen seen a tick earlier, so they are not independent. Confidence intervals are therefore cluster bootstrap intervals with the tick as the cluster:

• Clusters: the source.tick of each item (435 ticks for decisions; sheets and juries are asked at a smaller number of ticks, and a jury's votes always share a tick).
• Resample ticks with replacement, B = 10,000, keeping all items of each drawn tick; recompute the item-weighted mean.
• 95% percentile interval. Seed fixed at 20260921.
• Comparisons between systems (layatown-v1 vs floor, vs laya-base) are paired: the same resampled ticks for both systems, interval on the difference.
• Juror votes (168, few trials) will have wide intervals. That is reported as it is; no pooling with val.

5.6 Pre-registered thresholds (level A)

layatown-v1 (fitted temperatures) passes level A if all of the following hold:

A1 to A5 form one conjunction (intersection-union), so no correction for multiple comparisons is applied. For context, the val partials seen during training were TVD 0.114, top-1 0.865, sheet MAE 0.039; the thresholds are set looser than val on purpose, because the test day is a different, larger, live run.

6. Level B: human-labelled evaluation sets

These sets are small and some involve text typed by or about real people. They are evaluated locally only; rows are never copied into results files, only aggregate counts. Layatown was not trained on chat moderation, username screening or event reading: any skill it shows there comes from the base model.

The rows record the case and Jev's answer, not the full request. Each set is scored by rebuilding the request with the exact question wording from the benchmark scripts named in the data card (bench/chat.ts, bench/handles.ts, bench/shapes.ts, bench/trial.ts) and sending it to serve.py locally. Jev is represented by its logged answers.

Consequence of the gates: in --policy laya the city sends every request kind to Layatown, including chat, handle and proposal screening. A failed B1, B2 or B3 means Layatown must not serve that request kind in the live city (it goes to code-only screening or stays with Jev); the finding is reported and does not by itself fail the replacement question for citizen decisions.

7. Level C: city-level behaviour

7.1 Design

The simulator is deterministic given the world, seed and the decision-maker's answers (docs/ARCHITECTURE.md). All conditions share the engine, world data, perception, sampling (jev.floor, jev.stickiness), commitment rule and logging; only the decision-maker changes, exactly as in docs/EVALUATION.md.

• Scenarios: hard-week, siege, strange-week; world jevtown at hash a635a002c79cd306 (the runs must record this hash; a different hash invalidates the comparison).
• Seeds 1 to 5, 240 ticks, --fast, one run per scenario and seed with --policy laya pointed at a local serve.py running the frozen checkpoint. Run ids lt1-<scenario>-laya-s<seed>.
• Comparators: the existing eval6- / eval7- runs for jev, rules, random. No new Jev run. The ledger total (bun spend) is recorded before and after; any difference is a protocol violation and is reported.
• Sensitivity set: seeds 6 to 10 for laya and rules (both free), compared against the same Jev seeds 1 to 5. See 7.4 for why.
• Every Laya run must replay cleanly (bun replay runs/<id> verifies every tick hash).

7.2 Metrics

The existing bun compare metrics, unchanged, mean over seeds with 95% Student t intervals (n = 5): population at end, deaths, beatable threats overcome, injuries, median ticks until half the village knows, exhaustion collapses, thefts, friendships formed, quarrels, mean food stock, share asked per tick, share changing action per tick, action entropy, individuality (mean JSD to village), story beats, latency. Plus the bun report character contrasts per run (permutation p-values).

7.3 Pre-registered thresholds (level C)

Per scenario, layatown-v1 passes level C if all of the following hold:

7.4 What level C can and cannot show

All 15 eval6-/eval7- Jev runs are in the train split, so Layatown has been trained on Jev's decisions from exactly these scenarios and seeds. The matched comparison therefore tests whether imitation survives the closed loop (small per-decision errors compounding over 240 ticks into a different village), not generalisation. Trajectories diverge from Jev's within a few ticks because sampling and the world react to different answers, so the model soon sees perceptions that were never in training; but the early ticks are in-sample. The seeds 6 to 10 sensitivity set removes that overlap for the trajectory. If C1 to C4 pass on seeds 1 to 5 but fail on 6 to 10, the level C result is reported as failed.

7.5 The city (descriptive only)

The live city (jevcity) cannot be replayed under matched conditions: city-1 was run on world hash e05c5e46962d5688 and the current world hashes 07806946f4cc1ea5, and the production day depended on live players. One Laya city-week run on the current world is reported descriptively (population, deaths, per-action shares, latency), with no threshold.

7.6 Serving cost and latency (RQ5)

Measured on the RTX 3080 Ti, reported regardless, no pass/fail except one feasibility gate:

• From the level C runs: request latency p50 / p95 / max and decisions per second (the same summary.json fields as Jev's runs; Jev: p50 about 340 ms per village tick request, over the network).
• City load: 1,000 decisions per tick sent as 10 parallel requests of 100, states drawn from the val split (never test), cache off and cache on; p50 / p95 wall time per tick. Jev reference: 0.98 s and $0.015 per 1,000 decisions.
• Energy: GPU board power sampled at 1 Hz with nvidia-smi during the load test; reported as Wh per 1,000 decisions. No dollar figure is claimed for electricity or hardware.
• Cache hit rate (/health), because cached answers are exact but make the average look faster.
• D1 gate: p95 wall time for a 1,000-decision tick below the city's requestTimeoutMs (4,000 ms). Failing D1 means Layatown on this GPU cannot serve the full city at its current tick rate; it is reported, and the verdict below is qualified accordingly.

8. Verdict rules

• Faithful replacement: A1 to A5 and C1 to C4 (every scenario, seeds 1 to 5 and 6 to 10) all pass.
• Faithful judge, different city: level A passes, level C fails. Reported as such: per-question agreement does not carry to the closed loop.
• Not faithful: A1 or A2 fails, or C1 fails in any scenario where Jev passed. Reported as a failure, with the numbers, in the same place and at the same prominence a pass would have had.
• Level B gates decide which non-citizen request kinds Layatown may serve; D1 qualifies deployment.
• Reported regardless of outcome: every metric in sections 5 to 7 for layatown-v1, laya-base and floor, raw and fitted temperatures, dropped item counts, per-scenario tables, and the pre-plan exposures in section 9.

9. What was seen before this plan (disclosed exposures)

• Untouched Laya on 10% of the test day (results/base-raw-every10.json, every 10th item, raw temperatures): decision TVD 0.497, top-1 0.328 (n = 23,878); sheet MAE 0.360, side agreement 0.733 (n = 4,103); juror MAE 0.311 (n = 17). This was a look at the baseline before the plan existed. It did not inform any choice about layatown-v1's training data, hyperparameters or temperatures, which were set before it (except that the 256-token head length in model.py was chosen from token counts, not from scores). It is reported as a pre-plan look; the full-test laya-base numbers are recomputed under this plan.
• Test-day tokenization: layatown/cache/test_production-*.pt exist (token ids and Jev's targets). No fine-tuned model has been run on them.
• Validation partials during layatown-v1 training (step 2,000 of 4,345, every 20th val decision, every 10th val sheet answer): TVD 0.114, top-1 0.865, sheet MAE 0.039. Training runs a fixed schedule; there is no early stopping or checkpoint selection on these.
• Data card floors on the test day (TVD 0.517, top-1 0.278, juror MAE 0.112, sheet MAE 0.142) were computed by the dataset builder and are public in training/README.md.

10. Threats to validity

1. Fidelity is not correctness. Level A measures agreement with Jev, not with the truth. Distillation copies Jev's biases and errors along with its judgment: Jev's known failure modes (freezing when shown its own last answer, joining the crowd nearby, the neighbour effect in chat, the two-step inference it fails in famine, docs/JEV.md) are expected to transfer. A high score means "as good and as bad as Jev". Only level B touches correctness, on 445 rows.
2. The test day is not a new domain. It is a new moment of the same city world, with the same prompts, option wording, perception format and names style as train. It tests generalisation to new moments and people, not to new worlds or questions.
3. Context mismatch. Jev answered each citizen with up to 99 others in the same request; the dataset and serve.py give Layatown only the citizen's own slice (data card, "One item per example"). Level A inherits the dataset's slicing; level C inherits serve.py's. If the two slicings differ (for example in which top-level collections are kept, or option order), A and C measure slightly different inputs.
4. Temperatures fitted on val. Val is four live city runs, close in distribution to the test day. Fitted temperatures could be slightly optimistic for the city and less right for the village used in level C. Raw temperatures are reported for that reason.
5. List-position runs excluded. 407,653 decisions were dropped as wrong labels. The remaining data is later in the project, so it over-represents the post-fix world and the live city; the village runs are in train only.
6. In-sample closed loop. Level C matched seeds are in train (7.4). Mitigated by seeds 6 to 10.
7. Rounded, near-deterministic, merged labels. Jev's probabilities were logged to two or three decimals and duplicate requests averaged; a perfect student cannot score below the rounding noise, and Jev's own rerun variation (about 0.016 on juries) is a practical floor on MAE.
8. Small sets. 168 juror votes and 445 human-labelled rows give wide intervals; conclusions from them are directional.
9. Implementation drift. GPU inference in bf16 with batch-dependent padding is not bit-identical across batch compositions; runs remain replayable because decisions acted on are logged, but a rerun of a Laya run may not reproduce it exactly.
10. Licensing and terms. Laya is Apache-2.0 (inherited by the fine-tuned weights; attribution and licence notice required). The owner confirmed on 21 September 2026 that TypeSafe's terms permit training on these outputs (data card, "Provenance and permission"). Terms can change; check them before redistributing the weights or the data outside this project.
11. Same authors. The people who built Layatown wrote this plan and the thresholds. The thresholds are fixed here, before the test, to limit that.

10.5 Later variants

A variant trained after this plan (more data, more epochs, different weights per task) is chosen among others on val only. Each frozen variant is scored on the test day at most once, and every test score of every variant is reported, including the ones that lose. A variant designed after seeing layatown-v1's test results is labelled exploratory, and its test numbers are not used for the verdict of section 8 unless a new, untouched test day (a later production run, built by training/build.ts into a new split) is used for it.

11. Results

To be filled after the frozen run. Do not edit sections 1 to 10 to fit these.

11.1 Frozen checkpoint

11.2 Level A (test_production, 95% tick-bootstrap intervals)

Character effects (A5):

11.3 Level B

11.4 Level C (mean over seeds, 95% t intervals)

One table per scenario (hard-week, siege, strange-week), columns jev (eval6/eval7, s1-5), laya (s1-5), laya (s6-10), rules (s1-5), rules (s6-10), random; rows the metrics of 7.2; then C1 to C4 pass/fail.

TBD

Ledger total before / after level C: TBD / TBD.

11.5 Serving (RTX 3080 Ti)

11.6 Verdict

TBD (one of the three outcomes in section 8, with the failing criteria named).

Amendments

The plan above is left as first written. Changes are listed here with their date and reason.

• 2026-09-21, before the test day was scored: frozen checkpoint. checkpoints/v1/model.safetensors SHA-256 27f210cb321eda7e9dd93fd3c493d8c3b467e510353dbc4d013853801006429c. Temperatures fitted on validation (choice, score, noul): 1.0172, 1.0, 1.0014. Trained on 150,000 train decisions (random sample, seed 7), all 34,120 train sheet answers and the 251 train juror votes repeated 20 times; 4,345 steps, about 77 minutes on the RTX 3080 Ti. Last validation look (every 20th val decision, every 10th val sheet): decision TVD 0.107, top-1 0.866; sheet MAE 0.034. This checkpoint is the one scored; no training or tuning happens after the test day is opened.
• 2026-09-21, before any model was scored on the test day: floors recomputed with the section 5.4 rule. Decision floor on test_production: TVD 0.538, top-1 0.251 (the data card's 0.517 / 0.278 used a different averaging rule; thresholds use 0.538 / 0.251 as section 5.4 says). Sheet MAE 0.142 and juror MAE 0.112 match the card exactly. Computing the floor reads test targets but no model output; it was done after the checkpoint above was frozen.
• 2026-09-21: tooling changes before scoring. data.py now keeps each item's run, tick and ordered option set and counts dropped items (0 of 814,418 across all splits); evaluate.py gained 95% tick-bootstrap intervals (10,000 resamples), Brier scores and the floor. Character effects (A5), paired differences (A4) and level B/C need further scripts; they are written before those results are read.
• 2026-09-21, before any of it is measured: level D, "beyond the teacher". Fidelity to Jev (levels A and C) cannot show whether Layatown is better or worse than Jev, because Jev is the reference. Level D compares Jev, laya-base and layatown-v1 against references that do not come from Jev. Three suites, all built by code, all on test_production states (never train), fixed before any system's answers are read:
  • D1 written-rule compliance. Matched counterfactual pairs that differ in exactly one field of one person's perception, where the world's own criteria text states the direction (for example hunger starving vs fed with food available: p(eat) must rise; danger "right here" vs none: p(flee) + p(fight) must rise; trait cowardly vs brave: p(flee) must rise, p(fight) must fall; devout vs not: p(pray) must rise; a thief's job or trait vs not: p(steal) must rise). Score: share of pairs moving in the stated direction by more than 0.02, and mean signed effect. Only rules the criteria text states in words are used; the probe list and its source sentences are committed with the generator before any system is queried.
  • D2 invariance. The same item with its options in 3 other orders (and, separately, its state keys reordered). Score: mean TVD between orderings after mapping options back, and the share whose top choice changes. Lower is better; 0 is perfect.
  • D3 human labels. The five sets in training/data/eval, scored against the human label with each set's own rule (accuracy / agreement), using the same question wording Jev was asked. Jev's logged answers are used where they exist; nothing is re-asked of Jev that the logs already hold. Jev is queried only for D1 and D2 pairs, capped at 400 pairs and 100 items x 4 orders; projected cost under $0.25, ledgered through src/jev/client.ts. Sample sizes are fixed by seed before querying. A system "beats" another on a suite only if the paired 95% bootstrap interval of the difference (over persons for D1, items for D2, rows for D3) excludes 0. Reported regardless: all three systems on all three suites, including suites where Layatown is worse.
• 2026-09-21, before any system is queried for level D: how the level D amendment is implemented. Code in bench/beyond/ (generate.ts, rules.ts, common.ts, run.ts, d3.ts, analyze.ts); probes and results in bench/results/beyond/. Where the level D text is ambiguous, the most conservative reading is used, as follows.
  • Fixing the probes. generate.ts reads only training/data/test_production/villager_decision.jsonl (238,779 items) and drops the logged targets unread; seed 20260921. SHA-256 before any query: probes-d1.jsonl 71ea1d0ea94018e994b71d25267d55ca09ff795f86d67a40368f18bd56b09a0c, probes-d2.jsonl fe90a669c085ad165f85442e4318ccae64f33874e9669b5013d8ea16a87f054c, generate.ts bac15666c07335851a8b2741806fbc688fb8c980eee2337e3fb3830ab698e2b7, rules.ts ccef72af1150e0370cc8a88f4c6305aaf8d4e12a19dee8fa681f8266d88ff4b6, common.ts 68d669f7faebddce55f02c5705a9002b7793039937bfd520ab5ba2ce9315ca1d. Deviation: level D says the probe list is committed before any system is queried; the owner commits after the run, so these hashes, written here first, are the check that the committed probes are the ones queried.
  • D1, which rules. A rule is used only if the criteria text of the scored option names the manipulated condition (a value or a group) and says the option is what it calls for. Dropped from the pre-registered examples: devout -> pray and thief -> steal, because pray and steal have no description (null) in every test item, so the text states nothing about them. Also not used: night -> sleep and "no food" -> work (the condition is in village, not the person's perception, and food is never "none" on the test day), help others (not one field), cautious -> flee (stated; left out to keep the pre-registered cowardly vs brave contrast inside the cap). Used, 50 pairs each (distinct persons within a rule), 400 pairs in all, each differing in one field of one person: eat_starving hunger starving vs peckish, p(eat) up ("Get a meal. Comes first when starving."; "fed" read as peckish, the least hungry value the city ever offers eat, food available in every item); sleep_exhausted energy exhausted vs rested, p(sleep) up ("Comes first when exhausted"); danger_here the logged danger vs "none", p(flee) + p(fight) up, narrowed to wolves and bandits right here, the threats the fight text names ("Run to safety." / "Confront the threat and drive it off. ... wolves and bandits only leave when fought."); cowardly_vs_brave trait swapped in place, p(flee) up and p(fight) down, a pair complies only if both move by more than 0.02, and its signed effect is the mean of the two ("What the cowardly, the cautious, children and the wounded do." / "What guards and brave villagers do"); guard_fight job guard vs merchant, p(fight) up; child_flee job child vs merchant, p(flee) up (merchant: an adult job no criteria text mentions); wounded_flee health injured vs healthy, p(flee) up; work_healthy health healthy vs injured, adults, morning/midday/afternoon only, p(work) up ("The normal daytime activity for a healthy adult."). The generator refuses a probe whose criteria lack the quoted sentence.
  • D2. 100 items, one per person, drawn at random. Options: the logged order plus 3 distinct other orders. State keys: 3 other orders of the keys of every object with two or more keys (top level, village, the person); list values (traits, memories) keep their order. Per item: mean TVD over all 6 pairs of the 4 orderings (options matched by name), and whether the top choice differs in any of the 4 (ties broken by the logged option order, so a tie is never counted as a change). Jev is asked the option orders only: the cap "100 items x 4 orders" is spent there, so the state-key suite compares laya-base and layatown-v1 alone.
  • D3. Jev's logged answers are used, never re-asked (chat: the first of the two logged screenings; handles: the first score; event reading: the logged outcome; juror cells: the logged p). The local models get the historical wording: src/jev/screen.ts and src/jev/handle.ts (unchanged since those benchmarks ran), src/jev/prompt.ts with worlds/jevtown at commit e40fc25 and src/jev/shape.ts with worlds/jevtown at 7847d8c (both changed after their benchmarks; the code is extracted from git, not retyped), and the constants of bench/trial.ts (checked verbatim at run time). Scoring: chat, handles and event reading: accuracy against the human label with each benchmark's own rule (limits 0.4/0.5/0.5, 0.5, alsoOk, and "refused counts as right for a reject case"). chat_neighbour_effect has no per-row label beyond "harmless": primary score is the share still shown when screened beside the three abusive lines; the mean shift of the abuse score is secondary. juror_factorial has no human label: its reference is the pre-registered hypotheses H-T1 to H-T4 of bench/trial.ts, written by a person before any request; score = share of matched cell pairs in the hypothesised direction (368 pairs), bootstrapped over pairs; MAE to Jev is reported alongside. A request a local server refuses is scored as the benchmark code handles an unavailable model (fail closed) and counted.
  • Statistics. B = 10,000, seed 20260921, 95% percentile intervals, the same resampled units for every system (persons for D1, pooled across rules and within each rule; items for D2; rows or matched pairs for D3). Only units every compared system answered are used; failed requests are retried up to 3 times, then counted. Jev is jev-1.13.0; layatown-v1 is the frozen checkpoint at its fitted temperatures (the primary configuration); every request holds one person's state, shaped as in the data.
  • Cost. Dry count (characters / 4 plus 300 tokens per request, x 1.5): 1,200 Jev requests, about 752,000 tokens, projected $0.047, under the $0.25 cap. Ledger before level D: $16.6694.
  • Disclosure. While reading the criteria text of the test day, one logged Jev target (one calm dawn decision) was shown on screen. No other answer of any system was read before this amendment.
• 2026-09-21, before any fresh run is scored: level C gains out-of-sample Jev runs (owner-approved spend). The plan said level C needs no new Jev spend, which left the Jev side in-sample (every eval6/eval7 Jev run is in train). The owner approved about $3-4 for fresh Jev runs, so seeds 6-10 of hard-week, siege and strange-week are run on Jev with the eval6/eval7 settings (world jevtown, hash a635a002c79cd306, 240 ticks, --fast, $0.45 cap per run), named eval8-<scenario>-jev-s<seed>, batch capped at $4. Rules runs for seeds 6-10 are named eval8-<scenario>-rules-s<seed> and Laya runs lt1-<scenario>-laya-s<seed> for seeds 1-10. The PRIMARY level C comparison becomes seeds 6-10: jev vs laya vs rules, all out-of-sample for Layatown. Seeds 1-5 (in-sample Jev) are reported as secondary. The C1-C4 criteria are applied unchanged to seeds 6-10 with Jev's own seeds 6-10 as the reference. Scenario files are not hashed by the tooling; git log shows no change to scenarios/ since 2026-09-18, before eval6 ran.