Tessl's skill-eval study: skills help, but agents often don't load them

Tessl's research post reports a ~20-point accuracy gain from relevant skills, ~40% unforced activation, and leakage in ~30% of generated eval tasks.

Tessl's engineering blog post A Proposed Framework For Evaluating Skills (dated April 15, 2026, and recirculated this week by @bibryam) puts numbers on a question every skill author skips: does the SKILL.md actually change what the agent does? Tessl reports that access to a relevant skill gave an approximately 20% absolute accuracy gain over a no-skill baseline. It also found that agents activated available skills only about 40% of the time when not forced to, and that around 30% of its auto-generated eval tasks leaked scoring criteria into the task text, which it says can produce "misleadingly optimistic conclusions."

The method is paired runs. Tessl sampled skills from its registry, filtered out ones that need an existing codebase or infrastructure, generated up to five coding tasks per skill, then had separate reviewers check each task for criteria leakage, skill leakage and skill-specific value before comparing runs with and without the skill. These figures are Tessl's own and haven't been independently reproduced.

The warning matches what others report. A DEV Community write-up describes a skill that looked like a clear win on author-written checks but showed no gain on recall of planted flaws when a separate evaluator wrote new tasks. AgentPatterns.ai separates output quality from trigger precision and recommends testing each.

Tessl's CLI docs list a login or API key as a prerequisite. If you want an open-source route, NVIDIA's SkillEvaluator runs live with- and without-skill evals in a sandbox. Either way, the lesson is to measure activation and output separately rather than assume a skill works.

Sources