Open dataFirst dataset in progress · CC BY 4.0
How agents get Bun wrong
Probes run without skills to measure what agents do by default. The Atlas finds the mistakes, skills fix them, and the verifier proves it.
What the Atlas measures
Classified on every runhallucinated-apiCalls a Bun.* or bun:* member that doesn't exist in that Bun version.
node-fallbackImports a package from a skill's replaces list when the Bun API was available.
wrong-signaturebun check fails on a call to a Bun API.
deprecated-apiUses a symbol bun-types marks as deprecated.
runtime-failurePicks the right API, but the check test still fails.
Failure rates by API, the top mistakes and trends across Bun versions and models appear here once the first probe runs land. Monthly JSON and Parquet exports follow at /atlas/data.
Write a probe
probes/<api>/<id>.tomlA small, realistic prompt that should lead an agent to one Bun API. The probes are what the dataset is built from.
api = "Bun.SQL" id = "sqlite-notes" prompt = "In this Bun project, store notes in a local SQLite file and add list/create functions." fixture = "fixtures/empty-bun" expect_apis = ["Bun.SQL", "bun:sqlite"]