Lab / Before Beat the AI · 23 to 25 September 2026

Teaching the village new words, and checking the teacher

The village game gave its people parties, orders, trust and a loved one who can be dying. Our fine-tune of Laya, the model that ran the city, could not read those words. So we paid Jev, TypeSafe's System One model, to answer the new situations, trained new students on its answers in four rounds with the pass marks written in advance, and then audited the teacher itself. Two students qualified. Neither decided for players.

Study of 23 to 25 September 2026. Every round was pre-registered in the project's documents before it spent or trained: docs/LAYA-V3-ZEROSHOT.md, docs/LAYA-V2-PILOT.md, docs/LAYA-V2-FULL.md and docs/JUDGE.md, changed afterwards only by dated amendments. Every number below is copied from those documents and the result files they name. Teacher: Jev (jev-1.13.0) by TypeSafe. Base model: Laya by Convai Innovations (convaiinnovations/laya, revision 1c5edc17, Apache-2.0); every student here is our fine-tune of it, not Laya itself. Not affiliated with TypeSafe or Convai Innovations.

Could the city's model read the new words?

Before spending anything, we asked our city model, layatown-v1, unchanged, the exact questions the village game asks ($0, on our own GPU). Each check changes one word for one person and measures how far the answer moves; the bars were fixed before any request. It moved the right way on orders, but by a fifth of the bar or less. It did not read trust at all, and it went the wrong way on every one of 31 cases of a loved one dying.

The one v3 rule it did pass was guarding a loved one who lies gravely wounded (9 of 9 cases). On the held-out village games scored later, it gave Jev's top answer 0.618 of the time, and 0.440 for companions in a party. One cause we found: the instruction that names party, order and trust was cut by the model's input limit in 89% of the party questions. Note Jev's own T3: the teacher also treats "dying" as less urgent than "gravely wounded". That matters for everything below.

Four rounds, and what each cost

Each round was written down, with its budget cap, its stop rules and its pass marks, and committed before any of its requests or training runs. Candidates were chosen on validation data only; the held-out sets were opened once, for the chosen student. Training ran on one RTX 3080 Ti at home; only the teacher's answers cost money.

Every teacher game was rebuilt from its logs afterwards with no model calls; every tick hash matched. The full run cost within 0.7% of its written estimate.

The gates of C3 and F2

The gates were fixed before any v3 teacher data existed. G1: agreement with Jev on held-out games. G2: the same answer when the options are reordered. G3: written rules. G4: the one-word checks above, asked of the student: it must pass every check the teacher passed (G4a), and on every check move the teacher's way within 0.10 of it (G4b). G5: no loss on the city's held-out day. Both students are served averaging their answers over four option orders.

On the pilot's 66 held-out games, the set C3 was scored on, F2 reaches 0.865 against C3's 0.857. Its gaps to the teacher on the one-word checks: trust −0.026, hold the line −0.083, fall back +0.050, dying +0.006, a named tend order −0.058. The students still under-react to orders, now inside the tolerance.

How to read G2 and G3. G2 passes only because every answer is averaged over four orders; that costs about four times the GPU work, and a village request's 95th percentile is 1.10 s against the game's 1.5 s budget. G3 fails on "dying" because the teacher fails it (0.097): a student faithful to Jev cannot pass it, so we report it as inherited. F2 also falls short on the guard rule, 7 of 9 cases moving by more than 0.02, though all 9 move the right way; with 9 cases, one case is 0.11 of the score. The rule that a student replaces the stand-in when G1, G4 and G5 pass, with G2 and G3 reported, was recorded on 23 September before either student was scored, and confirmed by the owner on 24 September. Under it, F2 replaced C3 as the qualified model.

Checking the teacher

A student can only be as right as what it copies. So before training anything further, we re-asked Jev about its own training answers in controlled ways (a plan committed before any request; 36,000 requests, none failed, $2.1518). All figures are from docs/JUDGE.md section 11.

• Stability. 63.2% of the labels are stable, 36.0% shaky (moved by a reorder, mostly near-ties) and 0.79% unstable. Under 1% in every part, so by the written rule no labels were re-asked or dropped.
• Rerun noise against order. Jev is not bit-deterministic: 4.8% of identical requests change their top choice. The option order adds about 5.7 points on top of that. The city study below cautioned that Jev's 14% there includes rerun noise; here the two are measured apart.
• The context effect, 20.9%. The logged answers came from requests holding many villagers at once (about 52 in the village); a student learns from one person at a time. Asked about the same person alone, Jev changes its top choice 20.9% of the time (25.1% in the village), confidently in 10.5%. The students have been trained on answers shaped by a crowd they never see. This is the audit's one actionable result.
• Rule compliance. On 14 written-rule checks on its own training states, Jev complies 0.76 of the time overall. Strong on trust, orders and fleeing a threat here (0.91 to 1.00); weak on guarding a loved one who is down (0.60), sleeping when exhausted (0.54) and working when healthy (0.45); and against the rule on "dying" (0.14). A sentence that spelled the dying rule out for Jev turned the direction around, but not by enough, and it moved answers where nobody was dying, so by the plan's rule it was rejected and T3 stays inherited. On held-out rule pairs Jev scores 0.736: the mark a student would have to beat to be called better than its teacher on rules.
• An AI review, not a human one. Three separate Claude Opus 5.5 sessions, blind to each other and to the audit, judged 100 of Jev's decisions: is its top option a sensible next action? The panel said yes to 86 (95% interval 0.78 to 0.91), no to 3, unsure on 11, with Fleiss' kappa 0.72 among the three. A fourth judge, Claude Fable 5.1, agreed with the panel on 88 of 100. All three "no" answers fell on labels the audit had marked shaky. These judges are language models, not people; the review is a second opinion, not ground truth.

What this does not show

• One teacher. Every gate measures agreement with Jev jev-1.13.0. Agreement is not correctness, and the students copy the teacher's failures, as T3 shows.
• AI judges, not people. The only review of sensibleness was by AI judges, on 100 items. No human review is reported.
• No ethics board. No ethics board or outside reviewer has reviewed this work. The people who built the game wrote the plans and the pass marks, which is why they were fixed before the data.
• Simulated villagers. Every claim is about villagers in a simulation. The heroes, parties and orders in the training games were played by a script that cycles them on purpose, not by players, so they cover the words evenly and say nothing about how people play.
• No player data was used for training. Nothing anyone typed and no handles were ever in the dataset, and no v3 training game was played by a person.
• Our machine, not the cloud. Every latency here was measured on one RTX 3080 Ti. Nothing has been measured on the GPU the game would use.

What was next, on 25 September 2026

• The minds study. Does a small model retrained each night on the duel rounds players choose to share predict their next move better than the same model frozen, and better than a count of each player's habits? The groups, the gates, the minimum of 100 players, the 60-day hard stop and what happens to the rounds are in the plan. (Update, 26 Sep 2026: pre-registered and started that day, before any rounds were recorded.)
• The companions on the qualified model, through a cloud GPU. Only the hero's party (up to three companions) would ask F2; everyone else would stay on the game's rules, and each villager's card would say which decided. Before a switch-on: the cloud copy's answers checked against ours on the committed sets, latency measured from the game's server, hard caps on GPU use, and a privacy note naming where the model runs. Written for the village game; not started, and Beat the AI's duels do not use F2.
• Hob, a browser-sized student. Pre-registered in docs/MINI.md before any label or training: three sizes of about 6M, 17M and 50M parameters, starting from Ettin encoders (Johns Hopkins, MIT licence), learning from the odds of our qualified fine-tune (weight 0.7) and from Jev's answers weighted by the audit's stability tiers (0.3). Each size must pass G1, G4 and G5 against Jev and browser gates written in advance (at most 30 MB to download, latency on a throttled phone-like CPU, memory, frame rate, quantisation parity). The smallest size that passes is selected; if none does, that is published as the result. No Jev spend. Hob is neither Laya nor Jev, and its weights stay internal until the Season 1 release decision.
• The context effect. A round that asks Jev about each person alone, the student's own input (about $4.8 at measured prices), would address the audit's main finding. It needs its own plan and the owner's yes; it has not started.

The files

The project's repository is private for now; these names let each number be traced when the study files are published.

• Zero-shot check: docs/LAYA-V3-ZEROSHOT.md section 8, bench/results/laya-v3-zeroshot/
• Pilot, r2 and r3 (C3): docs/LAYA-V2-PILOT.md sections 4 and 9 to 13, bench/results/laya-v2-pilot/, laya-v2-r2/, laya-v2-r3/
• Full run and F2: docs/LAYA-V2-FULL.md sections 7, 11, 15 and 16, bench/results/laya-v2-full/
• Judge audit: docs/JUDGE.md section 11, bench/results/judge/
• Hob: docs/MINI.md
• Training data card: training/README.md

How it was made

Laya reads a situation and a question with fixed answers and returns a probability for each answer in one pass, the same shape as Jev. We trained it to match Jev's full probabilities (not just its top pick) on 150,000 decisions logged in the village and the city (18 to 20 September), 34,120 character-sheet answers and 251 juror votes: one epoch, 4,345 steps, about 77 minutes on the RTX 3080 Ti. The test is the city's first public day, 238,779 decisions, 41,024 character-sheet answers and 168 juror votes that it never trained on. Details are in the model card.

Does it give Jev's answers?

On the held-out day, with 95% intervals from resampling whole ticks. The floor always answers the average of training; "untouched Laya" is Convai Innovations' model with no fine-tuning.

* Untouched Laya was measured before the plan, on every 10th item of the same day with raw temperatures (23,878 decisions, 4,103 sheet answers, 17 juror votes), not on the full set. ** The paired interval the plan names: Layatown's error minus the floor's is −0.051 [−0.058, −0.043] (bootstrap over 14 trials, layatown/results/v1-juror-paired.json).

The character effects are the difference in how often a group does something compared with everyone else, on the same moments. Each must land within the larger of 25% of Jev's effect or 0.05:

Is it right where Jev is not the judge?

Agreeing with Jev cannot show whether a model is better or worse than Jev. So we also scored all three against references that do not come from Jev: rules the city's own text states in words, answers that should not change when the options are listed in another order, and small sets a person labelled. Better means the paired 95% interval of the difference excludes zero.

Two cautions on the wins. Jev is not bit-deterministic (2 to 7 of 50 choices change between identical calls), so its 14% includes rerun noise that no baseline separates out. And the rule check counts direction only: where Jev's effect is larger (cowardly against brave, children fleeing), Layatown complies more often but moves less. The screening sets are tasks Layatown was never trained on; it does them at or below the level of untouched Laya.

Does a village run on it live the same life?

Per-question agreement need not survive the closed loop, where small differences compound over 240 ticks. We ran the village of 80 through three scenarios on seeds 6 to 10, new for every system: 15 fresh Jev runs ($2.70), 15 Layatown runs and 15 runs of the hand-written rules.

On the 15 story metrics per scenario (deaths, injuries, thefts, friendships, food, exhaustion and so on), the 95% intervals of Layatown and Jev overlap on 13, 14 and 13 of them. That is a description, not a test of equivalence. Where they do not overlap: mean food stock in hard-week (Jev 186, Layatown 268), the share of citizens changing action each tick in hard-week and strange-week (Layatown slightly lower), and individuality in siege and strange-week (Layatown slightly higher). Strange-week produces only 2 to 6 threatened brave villagers per run, so C2 fails there for the reference system as well; the effect was positive in all ten runs. It is reported as a failure of the criterion as written.

Can one GPU keep up?

The GPU is the limit: about 140 decisions a second. Jev answered a full city tick of 1,000 in 0.98 s at the median; Layatown uses most of the 8-second tick.

What failed, plainly

1. The brave fight too much. On the held-out day Layatown's brave-fight effect is 0.268 against Jev's 0.180, outside the 0.05 tolerance. Criterion A5 fails as written.
2. One character test could not be passed by anyone. In strange-week, "the brave fight" reaches p < 0.01 in 0 of 5 seeds for Layatown and 0 of 5 for Jev. Criterion C2 fails there as written.
3. Our fine-tune reads people's words far worse than Jev. It was trained only on the city's decisions, and it lost ground even against Laya as released (chat 0.895 for untouched Laya). Chat 0.737 against 0.974, usernames 0.778 against 1.000, event proposals 0.426 against 0.987, and it blocked all 20 harmless lines placed beside abusive ones (Jev showed 17). That is why free text is gone: in Layatown no model reads what players type. Chat is a set of preset reactions, new events come from a menu in the world's data, and X handles that were never screened are not shown.
4. Juries follow their design less closely. 0.677 of matched jury pairs move the way the jury design predicted, against Jev's 0.910. Juries in the city now come from this model.
5. The serving margin is thin, and our own serving gate fails. The plan required a 1,000-decision tick to finish within 4 s at the 95th percentile. With Jev's 4 s limit, 3 of 60 ticks were skipped in the first scale test. We raised Layatown's limit to 7 s (the tick minus one second); at real speed its 95th percentile is 5.67 s. These tests ran on the same machine as the model; the live city adds the connection between Caracas and the server.

Verdict, by our own rules. The plan defines a "faithful replacement" as every criterion passing. Layatown is not one: A5 and C2 (strange-week) fail, and the serving gate is not met as written. It gives Jev's top answer 87% of the time on a day it never saw, keeps four of five character effects, keeps the village alive and as individual as Jev's, follows the written rules more often than Jev, and, because it was trained only on the city's decisions, is clearly worse at everything that involves reading what a person wrote (a limit of our fine-tune, not of Laya: the untouched model did better than our copy there). We switched with those trade-offs, and removed the features that depended on the last one.

What this does not show

Agreement with Jev is not correctness: Layatown copies Jev's known failure modes along with its judgment (the ones listed further down this page). The held-out day is a new moment of the same world, not a new world. The human-labelled sets are small (20 to 148 items). The village comparison is on 80 people, not the city of 1,000. The people who built Layatown wrote the plan and its pass marks, which is why they were fixed before the test.

Can a model teach itself? Round 1

Layatown learned everything from Jev. Can it get better without Jev, from signals that do not come from its teacher, and without getting worse at what it already does? Round 1 tried the cheapest such signal: self-consistency. The model was asked the same question four equivalent ways (the options in other orders, the state's keys in other orders), its four answers were averaged, and a copy of it was trained to give that average from a single view. No Jev label was used and no Jev request was made. The plan and its pass marks were committed before the first label was generated, and amended only in writing before training and before evaluation.

Data

100,000 decision states (50,000 from the model's own runs, 50,000 from the training split with Jev's answers thrown away), plus 34,120 character-sheet answers and 251 juror votes; 537,484 forward passes to label them

Training

From v1, lower 20 of 28 encoder layers frozen to share the GPU with the live city; lr 1e-5 / 3e-5, one epoch

GPU time

57.8 minutes to label, about 45.6 minutes to train (a bug in our own safety guard paused it for 29.5 more)

Pass rule

An improvement only if S1 (more consistent), S2 (no loss of fidelity) and S3 (no loss of rule-following) all hold

Verdict: a trade-off, not an improvement, so it was not deployed. The model did become slightly more consistent, but the gain was small (option-order TVD down about 5%) because v1 was already nearly invariant; there was little left to remove. It kept its fidelity and its rule-following. And its juror votes got worse, by 0.012 against a margin of 0.01 set before training: the four views of a juror's case disagreed on the verdict in 41% of the training cases, averaging turned those into hedges, and the student learned to lean toward 0.5 (mean p(guilty) 0.500, against v1's 0.494 and Jev's 0.479). The plan had named this risk: averaging blurs sharp answers.

Everything else stayed within noise of v1, including the brave-fight overshoot, which self-1 kept (0.267). The four village runs we made of each model (three siege seeds, one survival run) look alike, and are far too few for inference. Round 2 will take its signal from what happens to villagers in the simulation, and it gets its own plan before it runs.

Plan and amendments: layatown/docs/SELF-IMPROVE.md · results: bench/results/self-1/results.md, self-1-summary.json

What we are building next

Superseded: this section was written on 22 September, for You vs Laya, which never opened to the public. What came next is at the top of this page, and the study now is the minds study.

Two pieces of research, in the order they depend on each other. Neither exists yet; this is the plan, and each part gets its pass marks written before it is measured.

A dataset of people and a model in identical situations

In You vs Laya, every day a player finishes is replayed with the same luck and the model deciding in their place. That gives, for each moment, a person's choice and the model's choice in exactly the same situation, and what happened after each. Comparisons like that, with outcomes, are rare. Only with the player's consent, off by default, and without handles or names: the situation, the options, the choice and the outcome.

Nightly training, with gates

• Weighted by outcomes. Decisions that worked teach more, so bad play on purpose cannot steer the model.
• Frozen versions behind pre-registered gates. A candidate trains when the GPU is freest and is tested on the same suites as today's model, plus how it scores against players' logged days. It ships only if it passes; every version is dated in a public changelog, regressions first.
• Never retrained mid-session. A training run takes 45 to 80 GPU-minutes on the GPU that serves the game; online learning from anonymous players can drift or be poisoned; a model that changes all the time cannot be tested; and scores against an opponent that changed mid-run cannot be compared.

What we do not know yet. Whether people beat the model more often than not. Whether outcome weighting helps or just adds noise: round 1 shows a model's own signal can make one thing worse while making another better. And capacity: a player-day runs the day twice, estimated (not measured) at 8 to 16 GPU-seconds. We will measure before we promise.

The recipe we followed for v1 is written up for anyone to use with their own teacher: Train your own Laya. What it could be used for, with its limits: Build.

The files

The documents are rendered on this site from the project's repository; the result files are copied as they are and open as plain text.

• Plan and amendments: layatown/docs/EVALUATION.md
• Model card: layatown/docs/MODEL_CARD.md
• Held-out day: v1-test_production.json, v1-character.json, base-raw-every10.json
• Beyond the teacher: bench/results/beyond/results.md
• Village runs: bench/results/levelC/results.md, levelC-primary.txt, levelC-character.txt
• Serving: bench/results/serving/results.md
• Training data card: training/README.md
• Self-improvement, round 1: layatown/docs/SELF-IMPROVE.md, bench/results/self-1/results.md, self-1-summary.json
• Serving logs and power readings: lt1-realtime.log, lt1-failure.log, power-realtime.csv, soak-resources.csv