I Just Deleted All My Agent Skills
Key takeaways
- In one paired benchmark, 92 of 98 public software engineering skills had no significant effect on coding task results.
- A web development benchmark found that public skills lowered mean pass rates and raised token cost by 72% to 394%.
- Curated skills raised average pass rates from 33.9% to 50.5% in SkillsBench, so the evidence does not condemn every skill.
I just deleted all my agent skills after reading Jakob Breu’s (hot) take. I kind of had a gut feeling myself for a long time as models evolve but I didn’t really have the evidence to back it up, so I went looking for it. Some of what I found:
- SWE-Skills-Bench tested 98 public skills. 92 had no significant effect, and the average pass ratio went from 76.6% to 76.9%.
- Web development skills made results slightly worse and cost 72% to 394% more tokens.
- AGENTS.md files fare little better: no general gain in task success, over 20% more cost.
Also, Anthropic cut over 80% of Claude Code’s system prompt with “no measurable loss” on its own evaluations, and says it was overconstraining the agent, skills included. Karpathy’s prompts keep getting shorter. Boris Cherny reportedly says to delete the lot every six months.
Then there’s security. There seems to be at least one vulnerability in about a quarter of 31,132 skills. A second study found similar problems.
I might regret some of this. SkillsBench found curated skills lifted pass rates from 33.9% to 50.5%. Jim Bennett’s line seems right to me: skills that repeat the docs can go, but no model can infer your own process and voice.
Anthropic says much the same, as I understand it.