The experiment: one flag, everything else frozen
We took 25 complete agent sessions from SWE-bench Verified: real software-repair work on production open-source codebases, where an agent reads a repository, runs tests, and iterates on patches over roughly a hundred turns per task. 2,745 turns in all. Then we replayed them token for token: no re-generation, no sampling, the exact recorded conversation of each turn resubmitted as-is. That design removes every confound except the one under test. The serve ran DeepSeek-V4-Flash (a 304-billion-parameter mixture-of-experts, ~12B active per token, FP8) on a two-unit NVIDIA DGX Spark tensor-parallel pair. This is hardware that sits on a desk, not in a datacenter.
Two runs. Recall off: every turn recomputes its entire prompt from scratch. Recall on: the growing conversation prefix is served from memory, and only the new tokens are computed. The off-arm’s counters confirm the control is clean (zero cache queries), and all 2,745 turns completed in both arms. Timing and cached-token counts come from the serving engine’s own metrics, not ours.
Why agents make this number so large
An agent’s conversation only grows. Turn one is a page; turn eighty is a book. On turn eighty-one, a model without recall re-reads the whole book to add a sentence. Summed over a session, nearly all the compute is spent re-reading text the model has already read. That is why 98.3% of the prompt tokens in this workload could be served from memory: the number measures the workload’s redundancy, not a property of any one model. On the same turns, four unrelated architectures spanning ~200× in total parameters measured 90.6–98.6%. The saving belongs to how agents work.
Why the desk multiplies it
The same replay on a datacenter B300 shows a 3.6× wall-clock difference: large, but far from 25.7×. The cache-hit fraction is nearly identical on both machines; what differs is what a recomputed token costs. A datacenter GPU burns through redundant prefill quickly; a desk-side box cannot. So the scarcer the compute, the more each remembered token is worth. That inverts the usual story about the edge. Recall doesn’t merely make small hardware tolerable for agents. It makes small hardware rational: the hardware class that benefits most from remembering is exactly the class that runs beside you.
What this does not show
Three honest limits. First, this is a token-exact replay: the right instrument for isolating recall, but it means the agent is not improvising new actions live. Second, it is one workload class (multi-turn software repair); other agent workloads will have their own redundancy profiles, though the four-architecture spread above suggests the profile is broad. Third, the 25.7× is a wall-clock ratio on this specific hardware pair; the number that travels across hardware is the cache-hit fraction, and the multiple scales with how expensive prefill is wherever you run.
Method & integrity notes
The caching mechanism itself is standard serving infrastructure; prefix/KV-cache reuse ships in every modern serving stack. The contribution here is the controlled measurement: what that reuse is actually worth on real agent workloads, isolated cleanly, and where it is worth most. Token-exact replay of 2,745 agentic turns (25 SWE-bench Verified trajectories), twice, with prefix caching off then on; one serve flag different, everything else identical. Serve: DeepSeek-V4-Flash FP8 on a 2× DGX Spark (GB10) tensor-parallel pair, the same deployed serve we use daily. On-arm: 2,784 s wall-clock, 98.26% of prompt tokens served from cache (rounded to 98.3% above). Off-arm: 71,416 s, with the inverted control clean at zero cache queries. 2,745/2,745 turns completed in both arms; all counters and timings read from the serving engine’s own metrics. Cross-architecture context (90.6–98.6% across four models, and the 3.6× B300 comparison) comes from the same replay protocol run on identical turns.
Sources
SWE-bench Verified · DeepSeek (DeepSeek-V4-Flash) · NVIDIA DGX Spark