SkillsBench: Benchmarking How Well Skills Work Across Diverse Tasks
By Jeremy ·
Every agent framework worth using now ships with "skills": folders of instructions, scripts, and reference material an AI agent loads before it starts a task. The idea caught on fast. Almost nobody had measured whether skills actually change the outcome on real work, as opposed to demo tasks picked to look good. SkillsBench is the first serious attempt at that measurement, and the paper behind it has 77 authors, which tells you how many teams wanted their name on the answer.
What It Actually Tests
SkillsBench runs 87 tasks spread across eight professional domains: software engineering, finance, cybersecurity, mathematics, natural science, industrial systems, office work, and media production. Every task ships twice, once with a curated skill package attached and once without, so the same model and the same harness solve the identical problem under both conditions. The benchmark currently covers 18 model-harness configurations, pairing models like GPT-5.5, Claude, Gemini, and DeepSeek with agent harnesses such as Claude Code, Gemini CLI, and OpenHands.
That three-layer setup, skills, the harness running the agent, and the model underneath, is the whole point of the project. A skill only counts for something if it changes what a model actually does with a job, not what it claims it would do.
The Numbers
Across those 18 configurations, curated skills raised the average pass rate from 33.9 percent to 50.5 percent. That's a 16.6 point jump, a 25.5 percent gain over the no-skill baseline.
That's a real effect, not a rounding error.
It's also not an even one. Individual configurations gained anywhere from 4.1 to 25.7 percentage points. Some model-harness pairs barely noticed the skills were there. Others nearly doubled their pass rate. The paper's other finding backs up something agent builders have suspected without proof for a while: skill packages with three modules or fewer beat larger, more exhaustive bundles, and a smaller model with the right skill can match a bigger model working without one.
How It's Built
Each task in SkillsBench is a BenchFlow package: a task.md file with YAML metadata and the agent-facing prompt, a Docker environment, an oracle solution called solve.sh that proves the task is solvable at all, and a verifier that scores the result without a human in the loop. Running the getting-started setup requires Docker, Python 3.12 or newer, and the uv package manager, and the whole thing is open source under an Apache 2.0 license.
The oracle solution matters more than it sounds. A benchmark task with no known-working answer is just a guess about what "solved" means. SkillsBench checks its own homework before it asks an agent to.
Who It's For
SkillsBench isn't aimed at someone shopping for a chatbot. The audience is people building agent harnesses and skill libraries who need a number to argue with, plus teams deciding whether the latest "skills marketplace" is worth wiring into a production pipeline instead of just another folder of untested prompts.
What It Doesn't Tell You
The SkillsBench homepage leads with a different number than the paper does. It advertises GPT-5.5 going from 51.5 percent to 67.3 percent with skills turned on, a 15.8 point jump for one model. The paper's own aggregate, across all 18 configurations, is 33.9 to 50.5. Both numbers are real. They're just not the same claim, and a homepage leading with its best single result instead of its own average is worth remembering the next time a benchmark shows up in a pitch deck.
The range problem doesn't go away either. A configuration that gained 4.1 points and one that gained 25.7 points both get folded into that clean 16.6 point average. If you're deciding whether skills are worth building for your specific model and harness, that average tells you a lot less than it looks like it does.