Study: cutting coding-agent tokens with compaction can make runs 20–80% slower

A ~35,000-run arXiv study finds token-saving context compaction can slow coding agents. For anyone tuning Claude Code, OpenCode or Codex compaction.

Picture a two-person consultancy that runs coding agents overnight against client repos. A typical job is a long terminal session: upgrade a build, chase a failing test suite, patch it. The engineer who owns the harness watches one number, tokens per task, because that is what shows up on the invoice. So she tightens compaction. It triggers earlier, removes more history and summarizes harder. Token counts drop, and she counts it as a win.

Then the overnight runs start finishing later. A few tasks that used to pass now fail. Nobody connects either change to the compaction tweak, because the dashboard only plots tokens.

A new arXiv paper, Beyond Token Savings: A Systematic Study of Context Compression in LLM Agents (submitted 2026-09-26), puts numbers on that blind spot. Across nearly 35,000 agent runs, the authors report that "fewer tokens need not mean faster or cheaper execution." On Terminal-Bench with Qwen, policies using roughly one-third as many tokens can take 20–80% longer than the uncompressed agent. This is a measurement study, not a product. What you take from it is a different metric to track.

What it is

The paper splits a compression policy into three decisions that existing harnesses usually bundle together: how to compress, when to trigger it and how much to remove. It then varies each one separately across three open-weight models on SWE-bench Verified and Terminal-Bench 1.0. For each run it measures task success, token use, end-to-end latency and estimated cost (abstract).

Here is how the reported findings map to the pain in the lede:

The full HTML text groups mechanisms into two kinds. One kind discards older interactions. The other summarizes them with extra LLM calls, which the authors note "can themselves add token and latency overheads." The paper also cites a trace of 13 million GitHub Copilot sessions from separate prior work (Liu et al., 2026). In that trace, sessions that needed compaction accounted for 44.2% of tokens served, and the median compaction event removed 72.8% of the existing context. Those figures belong to the cited study, not to this paper's experiments.

The most detailed public summary so far is from @omarsar0, who attributes the work to UT Austin. He reports two comparisons:

He also says a policy that works well for Qwen drops Devstral to 38.7% and makes it slower. We could not confirm these per-policy figures in the fetched text, so treat them as his reading.

The knobs you actually have today, each described from its own source:

Who it's for

Pick this up if any of these apply:

Skip it if any of these apply:

Before and after

Before:

  1. Notice the agent bill or context warnings.
  2. Lower the compaction threshold or make it more aggressive.
  3. Check that tokens per task dropped.
  4. Ship the setting.

After (the paper's suggested approach, applied to your harness):

  1. Pick a small fixed set of real tasks from your own repos.
  2. Run them at your current setting and record tokens, wall-clock time, estimated cost and pass/fail per task.
  3. Change one decision at a time: trigger point, amount removed or mechanism.
  4. Compare per-task outcomes, not just the aggregate pass rate, because similar totals can hide different solved sets.
  5. Repeat when you switch models. The paper says results don't transfer reliably across models.

Try it

No code or data release was found, so you cannot rerun the paper's setup from the sources. The smallest real step is to instrument your own agent.

In Claude Code, according to Taku:

Run your fixed task set at two settings and time each run. The sources give no cost figure beyond your own API spend.

Where it breaks

Sources