The study / Jevtown System One dataset

Jevtown System One dataset

Rendered from training/README.md in the project repository, 21 September 2026, word for word. Laya is an open model by Convai Innovations (Apache-2.0); Jev is TypeSafe's model. Not affiliated with either. Cited result files are listed in the study.

Every judgment TypeSafe's Jev model made while running Jevtown, organised as training and evaluation data for building a System One model: a model that takes a situation and typed questions with fixed answers, and returns a probability for each answer, in one forward pass, without writing text.

Read this whole card before training on or evaluating against the data. It says what is exact, what is rebuilt, what is missing, and what the numbers can and cannot tell you.

At a glance

Plus five small evaluation sets with human labels in eval/ (445 rows). Everything fits in 512 tokens (median about 280). Built from 65 Jev-driven runs (38 in train, 4 in val, 1 held out as the production test), model jev-1.13.0, 18 to 20 September 2026. Figures above are from the build of 21 September 2026; data/stats.json has the current ones.

Build and check

bun training/build.ts      # reads runs/ (development) and runs/prod/runs/ (the production city), writes training/data/
bun training/validate.ts   # every line: schema, keys, valid probabilities, checksums, no personal data

training/data/ is gitignored: it is large and derived from paid model outputs. The scripts and this card are the source of truth; rebuilding from the same logs gives byte-identical files (checksums in data/manifest.json). The production logs are copied read-only from the city's volume: fly ssh console -a jevtown-play -C "tar czf - -C /data runs" > runs/prod/prod-runs.tgz then unpack into runs/prod/.

Format

One JSON object per line. Every example is one request in the systemone shape plus the answer it got:

{
  "id": "3f9c…",                         // sha256 of (task, state, questions), first 16 hex
  "task": "villager_decision",
  "state": { "village": { "time": "evening", "food": "scarce" },
             "villagers": { "Kara_Coldwood": { "job": "weaver", "traits": ["loyal","generous"], "hunger": "hungry",
                                                "danger": "wolves, right here", "memories": ["was robbed by Cora"] } } },
  "questions": { "c0001": { "type": "choice",
                            "instructions": "What does `villagers.Kara_Coldwood` do next? Judge from their `danger`, …",
                            "criteria": { "flee": "…", "help others": "…", "work": null } } },
  "targets":   { "c0001": { "type": "choice", "probabilities": { "flee": 0.62, "help others": 0.27, "work": 0.11 } } },
  "source":    { "run": "city-1", "origin": "development", "world": "jevcity", "world_hash": "e05c…", "tick": 212,
                 "model": "jev-1.13.0", "inputs": "exact", "shared_request_size": 100, "duplicates_merged": 1 },
  "flags": [],
  "approx_tokens": 281
}

• questions and targets have the same keys. choice targets carry probabilities over exactly the options in criteria (renormalised to sum to 1, three decimals). noul targets carry p, the probability of yes.
• criteria values are the option descriptions Jev was given; null means the option had no description.
• source.inputs is exact (the state is what was sent) or reconstructed_by_replay (see below).
• source.duplicates_merged: identical requests were merged and their answers averaged. Jev is near-deterministic (in the jury benchmark the same request sent twice moved answers by 0.016 on average), so this gives cleaner soft labels.
• approx_tokens is characters / 4 of state plus questions: a filter, not a tokenizer count.

The three tasks

villager_decision (choice, 8 to 11 options). What does this person do next? The state is one simulated citizen's perception (needs, mood, danger, traits, job, relationships, recent memories, news) and the village's time and food. Options are actions such as work, eat, sleep, flee, fight, help others, steal, pray, socialize, idle, with the decision rules written into the criteria ("Comes first when starving."). Exact: the logged perception is the object that was sent.

character_sheet (8 nouls per person). Would this person steal food if starving, risk their life for a friend, betray a friend for gold, stand and fight, forgive a thief, lie, lead, leave? The state is the person's job, traits and memories. Rebuilt by replay: the answers were logged; the state is recovered by replaying the run to the exact moment the question was asked. A sheet is kept only if the set of people alive at that moment matches the logged answers exactly.

juror_vote (noul). Does this juror vote guilty? The state is the case (charge, evidence) and one juror (job, traits, tie to the accused or victim, memories). Rebuilt by replay, with a check: every juror's tie, recomputed from the rebuilt world, must equal the logged tie, or the whole trial is dropped. None were dropped for a mismatch.

Splits

• test_production: the entire run of the city's first day in public (live-2026-09-19T23-51-37, 19 to 20 September 2026, 1,000 citizens, 435 ticks). Held out completely. This is the distribution a deployed model would face.
• val: about 10% of the other runs, chosen by a hash of the run id (whole runs, never mixed).
• train: the rest. Any request that also appears in val or test was removed from train (3,207 removed).

Split by whole runs, because consecutive ticks of one run are near-duplicates of each other.

What was left out, and why

• Runs that addressed people by list position (villagers[37]): 407,653 decisions. Jev cannot count; past about the twentieth item it answered about somebody else, so these answers belong to the wrong person. This is finding 16 in docs/JEV.md and "the expensive one" in the study. They are wrong labels and must not be used.
• Runs not driven by Jev: the hand-written stand-in policy, random, and fake models (1,083 runs). Their "probabilities" are not a model's.
• Anything a real person typed (chat messages, event proposals), X handles, and the X posts from the launch research. Privacy: players were promised their words are screened and shown, not collected into datasets.
• Two juries whose moment could not be found in the replay.

Evaluation sets with human labels (eval/)

Small, hand-written, from the project's pre-registered benchmarks (bench/, results in bench/results/). They test jobs other than the city and carry the correct answer written by a person (human_label) next to Jev's.

These rows record the case and the answers, not the full request wording; the exact questions are in the benchmark scripts (bench/shapes.ts, bench/chat.ts, bench/handles.ts, bench/trial.ts).

Baselines any model must beat (computed on this build)

The mean entropy of the decision targets is about 1.0 nat: Jev is usually decisive but rarely certain, so train on the soft probabilities, not just the top choice.

How to evaluate a model on this

1. Agreement with the teacher, per task: mean total variation distance and top-1 agreement (choice), mean absolute error and Brier score (noul), calibration error. Report the baselines above next to your numbers.
2. Behaviour, not just agreement. The study's pre-registered character effects are the real test: the cowardly flee (0.96 against 0.41), the brave fight (0.35 against 0.06), the devout pray (0.27 against 0.01), thieves steal (0.19 against 0.01). A student that matches Jev on average but loses these has learned the wrong thing. Run the city with it (bun sim … --policy <yours>) and compare with bun compare.
3. The known failure modes, so a new model does not repeat them: show it its own last answer (Jev: 80 of 80 froze), describe the crowd nearby (73 of 78 joined in), a harmless line next to an abusive one (0.41 against 0.09). docs/JEV.md has the details and the benchmark scripts reproduce them.
4. Speed and cost, measured on your hardware, next to Jev's measured 0.98 s for 1,000 decisions (ten parallel requests) and $0.015.

Known limitations

• The labels are a model's, not the truth. Targets are Jev's probabilities. A student trained on them learns to imitate Jev, strengths and weaknesses included. Only the eval/ sets carry human labels, and they are small.
• One item per example. In the city, Jev saw up to 100 citizens in one shared state (shared_request_size); each example here keeps only the item's own slice, which is how a small model would be asked. The answer was given with the others present, and context does move Jev's answers: in the chat benchmark a harmless line scored 0.09 for abuse alone and 0.41 next to an abusive one. How much the other 99 citizens moved each citizen's answer is not measured. A student asked about one citizen at a time may therefore learn slightly different answers than Jev would give in the same position; measure it before relying on exact agreement.
• Two worlds, one simulator. train has the village of 80 (jevtown, about 30%) and the city of 1,000 (jevcity), and test_production is the city. That tests generalisation to new moments and people, not to a new domain. A model meant for other jobs needs other data; the eval/ sets are a small first check.
• Rebuilt examples depend on replay. character_sheet and juror_vote states were recovered by deterministic replay (every tick's hash was verified). Replay stops at the first diverging tick; examples after it are not built.
• Rounded answers. Jev's probabilities were logged to two or three decimals.

Provenance and permission

• Collected by the Jevtown project (elberacasa), with every request recorded in the project's ledger.
• Model: TypeSafe Jev, jev-1.13.0. The owner confirmed on 21 September 2026 that TypeSafe's terms permit training on these outputs. Check the current terms before redistributing the data outside this project.
• The citizens are fictional: generated names, jobs and traits (worlds/*/villagers.json). No real person appears in the training splits; validate.ts rejects handles, links and email addresses.