Benchmarks

Conceptual

Standard suites, what they measure, and what they miss.

Benchmarks are useful for detecting regressions and comparing configurations. They are weak predictors of performance on your specific workload.

Categories we track#

Category	Measures	Known limitation
Software engineering	Resolving real repository issues	Skews toward well-tested open-source projects
Tool use	Correct selection and sequencing	Synthetic tool sets are cleaner than real ones
Multi-step research	Synthesis across sources	Answers may leak into training data
Long-horizon planning	Coherence over many steps	Few suites run long enough to matter
Robustness	Behaviour under adversarial input	Attack patterns age quickly
Calibration	Confidence accuracy	Rarely measured at all

We publish benchmark results with the harness configuration, budget caps and full traces, because a score without its conditions is not a measurement.

Last updated 2026-09-11