Most language model fine-tuning research solves a clean problem: take some text, predict the next token, collect a scalar loss, backpropagate, update weights. This works beautifully for single-turn tasks. But real AI agents don’t work this way. They make decisions across multiple timesteps. Environments branch into unexpected futures. Feedback arrives only after many steps, if at all. When researchers tried applying modern reinforcement learning to these long-horizon agentic tasks, they hit a wall: the computational overhead became prohibitive, and the theoretical advantages of RL seemed to dissolve under practical constraints.
A new paper argues for something counterintuitive. Instead of doubling down on RL, which seems like the obvious tool for sequential decision-making, go back to evolution strategies, a technique from the 1990s that most of the field moved past. On long-horizon agent tasks, ES actually outperforms RL for three concrete reasons: it fits larger models in memory, it composes naturally with prompt optimization, and it scales better as trajectory length grows.
The insight reframes how we should think about training agentic systems.
A quick word from our friends at The Control Plane…
The Control Plane
The systems behind AI are getting more complicated. The Control Plane helps you keep up.
The Control Plane covers AI infrastructure, agents, security, governance, and the technical systems needed to build and run AI in production.
Why single-turn fine-tuning breaks down
When you optimize a language model for a single-turn task, you’re navigating a direct pipeline from input to output. The model reads a question and generates an answer. Feedback is immediate. Gradients flow back through a single pass. This is why standard supervised fine-tuning and even basic RL work well in that regime.
Agentic reasoning is structurally different. Imagine an LLM controlling a web browser. It reads the current page, decides what to click, observes the next page, decides again. That sequence repeats for 15 steps. At each decision point, the agent’s choice creates branching possibilities: click button A and the site shows a login form; click button B and you’re redirected elsewhere. The reward signal is binary: task complete or task failed. There’s no per-step feedback about whether that click was good. The agent must assign credit across 15 decisions for a single outcome.
Agentic tasks branch exponentially and provide sparse rewards, making credit assignment fundamentally harder than single-turn prediction
This branching and sparsity expose RL’s architectural constraints. RL tries to solve credit assignment by decomposing the long trajectory into timestep-level contributions. It uses backpropagation to compute how each past decision influenced the final outcome. This is theoretically elegant but practically expensive. To backpropagate through a 15-step trajectory in a 27-billion-parameter model, you need to store intermediate activations for every layer at every timestep. GPU memory explodes. The gradient signal itself becomes noisy as you propagate backwards through 15 steps; the contribution of step 1 to the final reward is diffuse and unreliable by the time you compute it.
Worse, RL’s backprop infrastructure is tightly coupled. You’re committed to gradient-based updates. If you want to evolve the system prompt and the parameters simultaneously, you’re fighting the system. The tool wasn’t designed for that composition.
Why evolution strategies sidestep the problem
Evolution strategies ask a fundamentally different question. Instead of computing gradients, they sample perturbations. Take your current model. Generate small random modifications to its parameters. Run each modified model as an agent. Measure the reward. Update the model in the direction of high-reward perturbations.
This is black-box optimization. The algorithm never needs to know why a particular choice worked. It only sees the outcome. That simplicity unlocks three capabilities RL can’t provide.
-
Memory scalability: Backpropagation requires maintaining the full computational graph. Evolution strategies don’t. They only need to run inference on each perturbed model. No activation storage. No gradient computation. This means a 27-billion-parameter model uses only inference-level GPU memory, the same footprint as running it once to generate text. RL can’t achieve this because the backprop machinery is fundamental to how it works.
-
Flexibility: ES treats rewards as black-box numbers. The agent produces a trajectory, you assign a scalar score, the optimizer uses that score. This architecture composes naturally with prompt evolution. You can sample perturbations over both model parameters and system prompts simultaneously. The optimizer doesn’t care what’s being perturbed; it only sees rewards. RL’s gradient machinery doesn’t decompose this way. Prompt gradients and parameter gradients require separate optimization loops.
-
Long-horizon scalability: RL’s credit assignment problem worsens exponentially with trajectory length. At 5 steps, the gradient signals are clear. At 15 steps, noise dominates. At 50 steps, the approach breaks. Evolution strategies sidestep this entirely. They treat the whole trajectory as an atomic unit. A longer trajectory is just a “harder problem” with noisier gradients, but the optimization mechanism doesn’t fundamentally change. Horizons scale gracefully.
The algorithm: Agentic ESOpt
With the conceptual foundation set, how do you actually implement this for LLM agents? The algorithm is a simple loop.
Start with a base LLM. Sample a population of models by adding small random perturbations to the parameters. Each perturbed model acts as an agent in the environment, collecting a trajectory and receiving a reward signal. Normalize the rewards across the population. Update the base model parameters in a weighted sum of the perturbation directions, where weights come from the normalized rewards. Repeat.
More concretely: if perturbation A (adding noise in direction v1) leads to reward 0.8, and perturbation B (adding noise in direction v2) leads to reward 0.6, shift the parameters more in direction v1 and less in direction v2. The reward-weighted sum points toward promising parameter changes.
One design choice matters significantly: the magnitude of perturbations. Early in optimization, you want large perturbations to explore broadly and avoid getting stuck in local optima. Later, smaller perturbations for fine-tuning. The paper uses a cosine decay schedule for sigma, the perturbation standard deviation. It starts high and gradually decreases, encouraging broad exploration first, then focused refinement. This addresses the classic exploration-exploitation trade-off without requiring problem-specific tuning.
The algorithm is not sophisticated, which is part of its strength. It has few moving parts, minimal hyperparameters, and no tricks. That simplicity makes it robust.
Population sizing and backbone strength
Evolution strategies are known to be sensitive to population size. More samples should lead to better parameter updates, right? But the paper discovers something unexpected through experiments on a 15-turn Sudoku task. Stronger models need smaller populations to work well.
This makes sense if you think about the geometry of the perturbation space. A weak baseline model is lost in a high-dimensional parameter landscape. Many independent samples in different directions help find any promising direction at all. A strong model is already well-positioned. Most perturbation directions are local refinements pointing in similar directions. Extra samples just echo what you already know. A population of 8 beats a population of 16 when the backbone is strong because you’re wasting compute on redundant information.
This insight matters for practitioners. The standard assumption in ES research is “bigger population is better.” This paper suggests that assumption breaks when the base model is already good. For agentic fine-tuning with strong LLM backbones, smaller populations are not just acceptable but preferable.
Proof of concept on WebArena-Lite
Theory and intuition are necessary, but they don’t replace empirical validation. The paper evaluates Agentic ESOpt on WebArena-Lite, an environment where agents control web browsers to complete real tasks: find information on a website, fill out forms, navigate to specific pages. Tasks require 15+ steps of decision-making with binary feedback (task complete or not).
Full-parameter optimization of Qwen-27B improves the no-skill baseline by 6.69%. This is meaningful for long-horizon agentic tasks, where even 5% gains represent better environment navigation and decision-making. More importantly, it achieves this improvement while using only inference-level GPU memory on a 27-billion-parameter model. That’s the practical promise of the paper fulfilled: fine-tune large models for agentic tasks without prohibitive memory overhead.
The improvement is not transformative, but that’s expected. The baseline is already reasonably strong. The achievement is showing that modest gains are accessible with a method that scales where RL doesn’t.
This connects to related work on online reinforcement learning for multi-turn scenarios and resource-constrained agentic optimization, which also grapple with the tension between performance and computational constraints in long-horizon settings.
Test-time adaptation and prompt-parameter co-evolution
One of ES’s advertised advantages is flexibility. The paper demonstrates this by exploring test-time compute, where both the system prompt and model parameters evolve together during deployment.
Instead of fine-tuning offline and then freezing the model, Agentic ESOpt allows task-specific adaptation. For a new benchmark, start with the fine-tuned model, then evolve both the prompt and parameters toward that specific scenario. This is natural for ES because prompts and parameters are symmetric to the optimizer, both just “things to perturb.” The black-box reward signal guides their co-evolution.
The results show that Agentic ESOpt matches or beats baselines in 28 of 36 test-time settings. This is significant not for the raw win rate but for what it demonstrates: ES opens optimization possibilities that RL can’t easily support. Test-time adaptation of frozen parameters is awkward in RL because you’d need to recompute gradients. In ES, it’s a natural extension of the same algorithm.
This approach relates to ongoing research on automatic heuristic design for agentic systems and composable agent optimization, both exploring how to adapt agents beyond static fine-tuning.
What this teaches us about training agentic systems
The ML community often treats newer methods as strict improvements over older ones. Deep learning beat shallow learning. Transformers beat RNNs. RL beat supervised learning for sequential problems. This paper disrupts that narrative. Evolution strategies, a technique from decades past, beats modern RL for this specific problem class. The moral isn’t “ES was always better.” It’s “problem structure determines which tool wins.”
RL excels when you need fine-grained per-timestep credit assignment on short horizons with dense rewards. Evolutionary strategies excel when you need trajectory-level optimization on long horizons with sparse rewards and tight memory budgets. They’re tools for different purposes.
There’s a practical implication for anyone fine-tuning LLM agents. If you’re hitting GPU memory walls with standard RL approaches, if you care about composing parameter and prompt optimization, if your tasks have long horizons and sparse rewards, this paper offers a concrete alternative. It won’t feel like the “obvious” choice because the field converged on RL for sequential problems. But practical constraints often diverge from conventional wisdom.
Open questions remain. How far does ES scale on even larger models? Can you combine ES and RL, using ES for trajectory-level optimization and RL for within-trajectory refinement? Does the population scaling insight generalize beyond Sudoku and WebArena? These are research directions the paper opens.
For now, the takeaway is this: when your optimization problem requires long-horizon reasoning, sparse feedback, and minimal memory, sometimes the older approach wins.
