# Level D: beyond the teacher (results) Pre-registered in `layatown/docs/EVALUATION.md` (amendments of 2026-09-21). Generated by `bun bench/beyond/analyze.ts`. Every cell: estimate [95% paired bootstrap interval], B = 10,000, seed 20260921. A system is better only where the interval of the difference excludes 0. Probes: `probes-d1.jsonl`, `probes-d2.jsonl` (test_production only). | measure | n | jev | laya-base | layatown-v1 | paired differences | |---|---|---|---|---|---| | D1 compliance, all rules (share of pairs moving > 0.02 the stated way; higher is better) | 400 pairs / 337 persons | 0.895 [0.864, 0.924] | 0.203 [0.163, 0.243] | 0.968 [0.949, 0.983] | layatown-v1 - jev: 0.072 [0.044, 0.102] (**layatown-v1 better**)
layatown-v1 - laya-base: 0.765 [0.718, 0.809] (**layatown-v1 better**)
laya-base - jev: -0.693 [-0.742, -0.642] (**jev better**) | | D1 mean signed effect, all rules (higher is better) | 400 | 0.364 [0.338, 0.392] | 0.004 [-0.001, 0.009] | 0.361 [0.335, 0.387] | layatown-v1 - jev: -0.004 [-0.019, 0.012] (no difference shown)
layatown-v1 - laya-base: 0.356 [0.330, 0.384] (**layatown-v1 better**)
laya-base - jev: -0.360 [-0.388, -0.333] (**jev better**) | | D1 compliance: hunger starving vs peckish (food available): p(eat) must rise | 50 | 1.000 [1.000, 1.000] | 0.160 [0.060, 0.260] | 1.000 [1.000, 1.000] | layatown-v1 - jev: 0.000 [0.000, 0.000] (no difference shown)
layatown-v1 - laya-base: 0.840 [0.740, 0.940] (**layatown-v1 better**)
laya-base - jev: -0.840 [-0.940, -0.740] (**jev better**) | | D1 compliance: energy exhausted vs rested: p(sleep) must rise | 50 | 1.000 [1.000, 1.000] | 0.120 [0.040, 0.220] | 1.000 [1.000, 1.000] | layatown-v1 - jev: 0.000 [0.000, 0.000] (no difference shown)
layatown-v1 - laya-base: 0.880 [0.780, 0.960] (**layatown-v1 better**)
laya-base - jev: -0.880 [-0.960, -0.780] (**jev better**) | | D1 compliance: danger "wolves/bandits right here" vs "none": p(flee) + p(fight) must rise | 50 | 0.980 [0.940, 1.000] | 0.440 [0.300, 0.580] | 1.000 [1.000, 1.000] | layatown-v1 - jev: 0.020 [0.000, 0.060] (no difference shown)
layatown-v1 - laya-base: 0.560 [0.420, 0.700] (**layatown-v1 better**)
laya-base - jev: -0.540 [-0.680, -0.400] (**jev better**) | | D1 compliance: trait cowardly vs brave (same position in the list): p(flee) must rise and p(fight) must fall | 50 | 0.800 [0.680, 0.900] | 0.020 [0.000, 0.060] | 0.980 [0.940, 1.000] | layatown-v1 - jev: 0.180 [0.060, 0.300] (**layatown-v1 better**)
layatown-v1 - laya-base: 0.960 [0.900, 1.000] (**layatown-v1 better**)
laya-base - jev: -0.780 [-0.900, -0.640] (**jev better**) | | D1 compliance: job guard vs merchant: p(fight) must rise | 50 | 0.720 [0.600, 0.840] | 0.840 [0.740, 0.940] | 0.840 [0.740, 0.940] | layatown-v1 - jev: 0.120 [0.000, 0.240] (no difference shown)
layatown-v1 - laya-base: 0.000 [-0.140, 0.140] (no difference shown)
laya-base - jev: 0.120 [-0.020, 0.260] (no difference shown) | | D1 compliance: job child vs merchant: p(flee) must rise | 50 | 0.920 [0.840, 0.980] | 0.020 [0.000, 0.060] | 1.000 [1.000, 1.000] | layatown-v1 - jev: 0.080 [0.020, 0.160] (**layatown-v1 better**)
layatown-v1 - laya-base: 0.980 [0.940, 1.000] (**layatown-v1 better**)
laya-base - jev: -0.900 [-0.980, -0.820] (**jev better**) | | D1 compliance: health injured vs healthy: p(flee) must rise | 50 | 0.960 [0.900, 1.000] | 0.000 [0.000, 0.000] | 1.000 [1.000, 1.000] | layatown-v1 - jev: 0.040 [0.000, 0.100] (no difference shown)
layatown-v1 - laya-base: 1.000 [1.000, 1.000] (**layatown-v1 better**)
laya-base - jev: -0.960 [-1.000, -0.900] (**jev better**) | | D1 compliance: health healthy vs injured, adults, daytime: p(work) must rise | 50 | 0.780 [0.660, 0.880] | 0.020 [0.000, 0.060] | 0.920 [0.840, 0.980] | layatown-v1 - jev: 0.140 [0.040, 0.260] (**layatown-v1 better**)
layatown-v1 - laya-base: 0.900 [0.780, 0.980] (**layatown-v1 better**)
laya-base - jev: -0.760 [-0.880, -0.640] (**jev better**) | | D1 signed effect: eat_starving | 50 | 0.668 [0.612, 0.722] | -0.002 [-0.011, 0.006] | 0.716 [0.676, 0.753] | layatown-v1 - jev: 0.048 [0.018, 0.078] (**layatown-v1 better**)
layatown-v1 - laya-base: 0.718 [0.677, 0.756] (**layatown-v1 better**)
laya-base - jev: -0.670 [-0.724, -0.613] (**jev better**) | | D1 signed effect: sleep_exhausted | 50 | 0.665 [0.600, 0.725] | -0.006 [-0.015, 0.002] | 0.672 [0.627, 0.713] | layatown-v1 - jev: 0.007 [-0.029, 0.048] (no difference shown)
layatown-v1 - laya-base: 0.678 [0.631, 0.721] (**layatown-v1 better**)
laya-base - jev: -0.671 [-0.732, -0.606] (**jev better**) | | D1 signed effect: danger_here | 50 | 0.413 [0.354, 0.476] | 0.005 [-0.021, 0.031] | 0.423 [0.380, 0.467] | layatown-v1 - jev: 0.009 [-0.042, 0.061] (no difference shown)
layatown-v1 - laya-base: 0.417 [0.366, 0.471] (**layatown-v1 better**)
laya-base - jev: -0.408 [-0.479, -0.339] (**jev better**) | | D1 signed effect: cowardly_vs_brave | 50 | 0.335 [0.279, 0.393] | 0.002 [-0.002, 0.007] | 0.275 [0.236, 0.318] | layatown-v1 - jev: -0.060 [-0.103, -0.017] (**jev better**)
layatown-v1 - laya-base: 0.273 [0.233, 0.316] (**layatown-v1 better**)
laya-base - jev: -0.332 [-0.390, -0.277] (**jev better**) | | D1 signed effect: guard_fight | 50 | 0.218 [0.155, 0.287] | 0.061 [0.046, 0.077] | 0.195 [0.144, 0.251] | layatown-v1 - jev: -0.023 [-0.058, 0.012] (no difference shown)
layatown-v1 - laya-base: 0.134 [0.088, 0.182] (**layatown-v1 better**)
laya-base - jev: -0.157 [-0.220, -0.099] (**jev better**) | | D1 signed effect: child_flee | 50 | 0.238 [0.187, 0.291] | -0.006 [-0.008, -0.003] | 0.155 [0.130, 0.183] | layatown-v1 - jev: -0.083 [-0.140, -0.026] (**jev better**)
layatown-v1 - laya-base: 0.161 [0.135, 0.189] (**layatown-v1 better**)
laya-base - jev: -0.244 [-0.296, -0.194] (**jev better**) | | D1 signed effect: wounded_flee | 50 | 0.224 [0.183, 0.268] | -0.003 [-0.005, -0.001] | 0.192 [0.164, 0.223] | layatown-v1 - jev: -0.032 [-0.070, 0.004] (no difference shown)
layatown-v1 - laya-base: 0.194 [0.167, 0.226] (**layatown-v1 better**)
laya-base - jev: -0.227 [-0.270, -0.186] (**jev better**) | | D1 signed effect: work_healthy | 50 | 0.154 [0.112, 0.200] | -0.019 [-0.028, -0.011] | 0.258 [0.202, 0.316] | layatown-v1 - jev: 0.103 [0.069, 0.140] (**layatown-v1 better**)
layatown-v1 - laya-base: 0.277 [0.218, 0.337] (**layatown-v1 better**)
laya-base - jev: -0.173 [-0.221, -0.128] (**jev better**) | | D2 options: mean TVD between the 4 orders (lower is better) | 100 | 0.094 [0.084, 0.104] | 0.209 [0.199, 0.221] | 0.015 [0.014, 0.017] | layatown-v1 - jev: -0.078 [-0.088, -0.069] (**layatown-v1 better**)
layatown-v1 - laya-base: -0.194 [-0.205, -0.183] (**layatown-v1 better**)
laya-base - jev: 0.116 [0.101, 0.130] (**jev better**) | | D2 options: share whose top choice changes (lower is better) | 100 | 0.140 [0.080, 0.210] | 0.790 [0.710, 0.870] | 0.010 [0.000, 0.030] | layatown-v1 - jev: -0.130 [-0.210, -0.060] (**layatown-v1 better**)
layatown-v1 - laya-base: -0.780 [-0.860, -0.690] (**layatown-v1 better**)
laya-base - jev: 0.650 [0.540, 0.750] (**jev better**) | | D2 state keys: mean TVD between the 4 orders (lower is better; Jev not asked) | 100 | n/a | 0.196 [0.180, 0.212] | 0.053 [0.047, 0.059] | layatown-v1 - laya-base: -0.143 [-0.160, -0.126] (**layatown-v1 better**) | | D2 state keys: share whose top choice changes (lower is better; Jev not asked) | 100 | n/a | 0.680 [0.590, 0.770] | 0.040 [0.010, 0.080] | layatown-v1 - laya-base: -0.640 [-0.730, -0.540] (**layatown-v1 better**) | | D3 chat_moderation: accuracy against the human label, with the benchmark's own rule (higher is better) | 38 | 0.974 [0.921, 1.000] | 0.895 [0.789, 0.974] | 0.737 [0.605, 0.868] | layatown-v1 - jev: -0.237 [-0.368, -0.105] (**jev better**)
layatown-v1 - laya-base: -0.158 [-0.289, -0.026] (**laya-base better**)
laya-base - jev: -0.079 [-0.184, 0.026] (no difference shown) | | D3 chat_neighbour_effect: share of the 20 harmless lines (human label: show) still shown when screened beside three abusive lines (higher is better) | 20 | 0.850 [0.700, 1.000] | 0.000 [0.000, 0.000] | 0.000 [0.000, 0.000] | layatown-v1 - jev: -0.850 [-1.000, -0.700] (**jev better**)
layatown-v1 - laya-base: 0.000 [0.000, 0.000] (no difference shown)
laya-base - jev: -0.850 [-1.000, -0.700] (**jev better**) | | D3 username_screening: accuracy against the human label, with the benchmark's own rule (higher is better) | 36 | 1.000 [1.000, 1.000] | 0.833 [0.694, 0.944] | 0.778 [0.639, 0.917] | layatown-v1 - jev: -0.222 [-0.361, -0.083] (**jev better**)
layatown-v1 - laya-base: -0.056 [-0.139, 0.000] (no difference shown)
laya-base - jev: -0.167 [-0.306, -0.056] (**jev better**) | | D3 event_reading: accuracy against the human label, with the benchmark's own rule (higher is better) | 148 | 0.987 [0.966, 1.000] | 0.405 [0.324, 0.486] | 0.426 [0.345, 0.507] | layatown-v1 - jev: -0.561 [-0.642, -0.480] (**jev better**)
layatown-v1 - laya-base: 0.020 [-0.047, 0.088] (no difference shown)
laya-base - jev: -0.581 [-0.662, -0.507] (**jev better**) | | D3 juror_factorial: share of matched cell pairs in the direction of bench/trial.ts's pre-registered hypotheses H-T1..H-T4 (higher is better) | 368 | 0.910 [0.880, 0.940] | 0.543 [0.492, 0.595] | 0.677 [0.628, 0.723] | layatown-v1 - jev: -0.234 [-0.285, -0.182] (**jev better**)
layatown-v1 - laya-base: 0.133 [0.076, 0.188] (**layatown-v1 better**)
laya-base - jev: -0.367 [-0.424, -0.310] (**jev better**) | | D3 chat_neighbour_effect: mean |abuse score beside - alone| (lower is better) | 20 | 0.115 [0.073, 0.159] | 0.674 [0.633, 0.711] | 0.295 [0.264, 0.323] | layatown-v1 - jev: 0.181 [0.115, 0.241] (**jev better**)
layatown-v1 - laya-base: -0.379 [-0.396, -0.361] (**layatown-v1 better**)
laya-base - jev: 0.560 [0.482, 0.630] (**jev better**) | Requests and failures: jev 1200 sent, 0 failed (jev-1.13.0); laya-base 1500 sent, 0 failed (rl-agent-v1); layatown-v1 1500 sent, 0 failed (layatown-v1). D3 requests the local server refused (scored fail-closed, as the benchmark code does): {"chat_moderation":{"laya-base":0,"layatown-v1":0},"chat_neighbour_effect":{"laya-base":0,"layatown-v1":0},"username_screening":{"laya-base":0,"layatown-v1":0},"event_reading":{"laya-base":0,"layatown-v1":0},"juror_factorial":{"laya-base":0,"layatown-v1":0}}. Jev input tokens: 867516 ($0.0364), ledgered as "bench:beyond". Reading notes (not part of the pre-registered measures): - Jev is not bit-deterministic (docs/JEV.md: 2 to 7 of 50 choices change between identical calls); its D2 numbers include that rerun noise, and no noise-only baseline was pre-registered. Layatown is deterministic up to bf16 batching. D2 cannot say how much of Jev's 0.094 is order sensitivity and how much is noise. - D1 counts direction only. Mean signed effects show the size: where Jev's effect is larger (cowardly vs brave, child flee), Layatown complies more often but moves less. - juror_factorial is scored against the benchmark's pre-registered hypotheses, not against a human label per row. - The local models answer the D3 requests exactly as the historical code sends them, including the batched `messages[i]` request of the neighbour test, which serve.py cannot slice (it is not a `collection.key` path).