What we measured, and what it showed.
The same questions, answered by Claude alone and by Claude with Day Theory. Every gain is checked with an exact paired test. Where a gain is too small to prove, we say so.
How every number on this site is measured.
Plain rules, the same for every test.
- Same questions, both ways
- Every test asks the same questions of Claude alone and of Claude with Day Theory, so each question is its own comparison. Each question is run twice in each arm.
- The bar
- More right answers, under a 1 in 20 chance of luck, and no group of questions down more than 10 points. The bar is set before the test.
- Exact odds
- The chance a gain is luck is exact, counted from every way the runs could be relabelled. No approximations.
- Held-out and development sets
- Held-out questions were not used while the knowledge was written. Development sets are labelled wherever they appear: held out from the knowledge, but run repeatedly during development.
- Scored strictly
- An answer counts only if it is right. Every score is a floor.
- Cost
- Claude usage per right answer: measured usage divided by right answers, without Day Theory’s own price.
- Models
- Claude Sonnet 5.5 for software incidents, filings and documentation. Claude Opus 5.5 for Telecom, Bank and Market.
Numbers. Run counts and exact tests come from the site’s own data and statistics files, loaded unchanged: software incidents (133 held-out incidents on three systems), financial filings (150 questions, development set) and documentation (124 held-out questions on 13 public databases, with half or all of the documentation), each run twice with Claude Sonnet 5.5. The per-system rows and the 8-database row come from the numbers guide; Telecom, Bank and Market cost rows from the coordinator’s run record of 13 November; the warehouse runs from the graph gallery’s run data, loaded unchanged. Telecom ran an earlier version on Opus 5.5 and is a development set.
Self-check. Running…