Ground rules
Every experiment here follows these rules. I wrote them down before running anything, so you can hold me to them.
- I built the thing I’m testing. I wrote Sojourner, and I wrote the tasks, hidden tests and scoring, so I expect to be biased toward it. Every rule below exists partly to keep that bias in check. The plain setup I compare against has its prompts published, so you can judge whether it got a fair shot. Each post shares the evidence: the plan, the tasks, the hidden tests and a record of every run. The code that runs the experiments stays private, so you can check every number but not rerun the experiment. Instead, each expedition describes in plain words how runs work and how they’re scored, in enough detail to build your own and test the same question.
- The plan comes first. Before the first run, each experiment writes down what I expect to happen, the one number that decides it, how many runs it gets, and what result would count as “no effect.” If anything changes after that, the change is logged with its date and reason. The plan is published on this site before the first run, and each post says which version it followed.
- One change at a time. Each comparison changes exactly one thing against its baseline. If two things changed, there’s no telling which one mattered.
- Enough runs, set up front. The plan sets how many runs happen, not how much usage I have left: at least three runs per setup on every task, and about thirty scored runs per setup in total. Nothing is analyzed until that count is reached, and nothing stops early because it looks good. Posts report the counts and a 95% confidence interval, not just an average. If my usage budget can’t reach the count, the post says so as a limitation.
- Every run is kept. Failures stay in the record. A mistake the agent made counts as a fail. Runs lost to a broken harness or a usage limit are left out of the count, and how many there were is reported.
- Hidden tests stay hidden. The tests that score a run live outside the agents’ workspace and are only copied in after the run ends. I check the transcripts for any attempt to find them.
- Everything is pinned and recorded. Each run records the exact model versions, Claude Code version, Sojourner version, prompts, machine and usage. Cost is reported as what the run would cost at API prices, with the price list cited, and then as subscription usage.
- Tasks don’t change once they have results. If a task needs fixing, it gets a new version instead of an edit. Once a task’s hidden tests are published, that version is retired, so no later experiment is scored on tests anyone could have seen.
- No made-up numbers. Every number in a post traces back to a recorded run. Missing data is reported as missing.
- It has to help someone. Each experiment names who the answer helps and what decision it would change for them. If a finding wouldn’t change what anyone does, the post says so plainly.
Walk along
New writing, sent to your inbox when there's something worth sharing.