The Right Amount of Memory for Your Agent: A Study on Calibration and Dosage

The concept of equipping an agent with agentic memory seems straightforward enough. Distill lessons from its past work, put them back in context, and more experience should mean better performance. However, the reality is far more complex. When we scaled up our evaluation to eight models – ranging from a 30B dense model to frontier proprietary systems – one finding stood out: agentic memory is not a feature you switch on; it’s a dose that needs to be calibrated to the model.

In other words, not every agent benefits from the same amount of memory. Across these eight models spanning different capability levels, we observed three recurring patterns. The first pattern involves strong models with headroom wanting the full guideline set – including rare edge-case lessons. They have the capacity to absorb and apply all of it. For instance, DeepSeek-V3.2 (671B MoE) saw a +9.5 percentage point gain in task completion when given its full self-mined guideline set.

The second pattern involves smaller or weaker models getting overwhelmed by a large guideline set. These agents do best with a tight, high-confidence core plus a handful of task-relevant guidelines retrieved per task. gpt-oss-120b (117B MoE) gained +16.1pp in this selective approach – while the full guideline set resulted in less gain and cost around 50% more tokens.

The third pattern involves already-saturated models showing no measurable gain. We label this as the saturated pattern, describing what we observed rather than a proven cause. The model may have been near its ceiling on these tasks, or the guidelines might not have addressed its remaining failures. GLM-5 (745B MoE) sat in this category during our runs.

What determines whether an agent falls into one of these patterns isn’t just parameter count. Benchmark headroom, context-window size, architecture, guideline quality, and task distribution all seem to play a role – though separating those factors is ongoing work. The practical takeaway remains the same: the right dose of memory depends on the model, and we can calibrate it.

The concept of ‘memory’ here doesn’t refer to replaying past transcripts but rather a set of guidelines distilled from an agent’s own prior trajectories. These strategies worked, mistakes to avoid, and edge cases are consolidated into a reusable set that can be injected at inference time without updating model weights – which is why this approach is cheap to adopt and portable across the eight models tested.

We evaluated our findings on AppWorld, a rigorous multi-step benchmark consisting of 585 tasks (168 test_normal + 417 test_challenge) across nine simulated apps. Tasks are scored in two ways: whether an agent fully completes each task (TGC – Task Goal Completion) and whether every variant of a scenario passes (SGC – Scenario Goal Completion).

Because the confusing part of any memory study is what’s actually within the context window, we defined our configurations up front. Both memory configurations draw from the same guideline set mined once via the loop described above from AppWorld’s training split only. What changes between them is how that one set is delivered – with the full guideline set injecting all guidelines every step versus curated retrieval delivering a selected subset.

The number of guidelines an agent mines depends on its own capability, so we report configurations by strategy rather than raw counts to ensure comparability across models. Representative models from our eight-model sweep are plotted in Figure 1, measured by task completion (TGC) on test_normal – though the stricter SGC metric often shows larger gains.

The figure plots TGC for readability; however, the table adds the stricter SGC metric where the gains are frequently larger: DeepSeek’s SGC jumps +16.1pp against a +9.5pp TGC gain because good guidelines especially help an agent clear every variant of a scenario – not just the average case.

A practical concern is that injecting a full guideline set inflates each ReAct step’s input, as the guidelines are re-sent each turn. However, we observed two key takeaways: curated retrieval keeps cost near baseline for weaker models where selection wins on accuracy; and memory doesn’t blow up the reasoning loop – DeepSeek runs about the same number of ReAct steps with or without memory.

The real efficiency lever in production is prompt caching: the static portion of the guideline set can be cached, cutting effective cost substantially. Cache-aware prompt design keeps the shared guideline-set prefix stable so it stays cacheable – an engineering effort worth pursuing. We also hypothesize that context-window size plays a role: models with larger windows may absorb the full guideline set more effectively while smaller-context models benefit from retrieval keeping injected content compact.

The lesson isn’t to give an agent everything it has learned but rather to provide the amount of experience it can actually use – which varies greatly across agents. For weak models, this means a compact core plus a few task-specific lessons; for strong models with headroom, preserving the full guideline set is key; and for saturated models, spending no extra context until their remaining failure modes are better understood.

The gains from memory are real but only when the dose fits the model – automatic, leakage-free, and requiring no human annotation. This study marks a starting point rather than the finish line: we’re exploring learned selectors to improve retrieval rankings; memory for very weak models where self-distillation lacks signal; broader agent benchmarks and real-world deployments; and isolating context window size from raw capability.

Try the ALTK-Evolve library, which includes the extraction, consolidation, and retrieval pipeline used in this study – or read our full technical report for more details on method and ablations.