GEPA Prompt Optimization: Evolutionary Prompt Evolution for DSPy

What GEPA Is

GEPA (Genetic-Pareto) is a prompt optimizer from the paper "GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning" (Agrawal et al., 2025, arxiv:2507.19457), accepted as an oral presentation at ICLR 2026. It is integrated into DSPy as dspy.GEPA and also ships as a standalone library (pip install gepa).

The core problem GEPA solves: conventional optimizers collapse rich execution data into a single number. They know a prompt scored 0.4 out of 1.0, but they do not know why. Was it a formatting error? A reasoning failure? A missing constraint? Without this diagnostic information, the optimizer has to brute-force its way through the search space, trying random variations until something works.

GEPA takes a different approach. Evaluators return Actionable Side Information (ASI) alongside the score. ASI can be error messages, reasoning traces, profiling data, constraint violations, or any text that helps diagnose the failure. A strong LLM (the "reflector") reads this feedback, identifies what went wrong, and proposes a new prompt that addresses the specific failure mode. Each mutation inherits accumulated lessons from all ancestors in the search tree.

How GEPA Works

GEPA's optimization loop has five stages, repeated for each generation:

1. Candidate Selection from the Pareto Frontier

The optimizer selects a parent candidate from the Pareto frontier. Selection probability is proportional to coverage: candidates that are the sole best on many evaluation instances get sampled more often. This balances exploitation (improving strong candidates) with exploration (giving underrepresented strategies a chance).

2. Minibatch Evaluation with Trace Capture

The selected candidate runs on a small minibatch from the training set (default size: 2 examples). GEPA captures the full execution trace for each example: the prompt sent, the model's reasoning steps, tool calls and their outputs, the final answer, and why the evaluator scored it the way it did. The trace includes ASI returned by the evaluator.

3. LLM Reflection

A strong LLM (the reflection_lm) receives the execution traces, both successes and failures, and produces a diagnosis. The reflector identifies failure modes, logic breakdowns, and causal relationships. It sees what worked in successful examples and what broke in failures, building a complete picture of the candidate's strengths and weaknesses.

4. Targeted Mutation

Based on the reflection, GEPA proposes a new prompt that addresses the diagnosed issues. This is not a random mutation. The new prompt inherits all accumulated lessons from its ancestors in the search tree, plus the specific fixes identified by the reflector. GEPA also supports system-aware merge: combining the strengths of two Pareto-optimal candidates that excel on different problem types.

5. Pareto Validation

The new candidate is evaluated on the full validation set. If it achieves the best score on any evaluation instance, it joins the Pareto frontier. If an existing candidate is now dominated (beaten on every instance), it drops off the frontier. This ensures the frontier only grows with genuinely complementary candidates.

GEPA vs Other DSPy Optimizers

DSPy ships with eight optimizers covering different tradeoffs. GEPA occupies the high-quality, instruction-only end of the spectrum. Here is how it compares to the other major options:

BootstrapFewShot: The Starting Point

BootstrapFewShot generates demonstrations using a teacher model and filters them by your metric. It is fast, cheap, and requires minimal data. But it only optimizes few-shot examples, not the instructions themselves. For tasks where the instruction quality matters more than the examples (complex reasoning, math, agentic workflows), BootstrapFewShot hits a ceiling quickly.

MIPROv2: The Previous Best

MIPROv2 uses Bayesian optimization to search over both instructions and demonstrations. It is data-aware and demo-aware, producing strong results with enough data (200+ examples) and compute (40+ trials). GEPA outperforms MIPROv2 by 13% aggregate and by +12% on AIME 2025. The key difference: MIPROv2 treats the metric score as the only feedback signal. GEPA reads the full execution trace.

COPRO: Coordinate Ascent

COPRO generates and refines instructions via hill-climbing. It is simple and interpretable but lacks diversity preservation. Once COPRO finds a local optimum, it has no mechanism to escape. GEPA's Pareto frontier maintains multiple candidates simultaneously, each representing a different local optimum. The merge operation can combine their strengths.

SIMBA: Failure-Focused

SIMBA identifies examples with high output variability (where the model sometimes succeeds and sometimes fails) and uses LLM introspection to generate improvement rules. GEPA and SIMBA share the idea of LLM-driven diagnosis, but GEPA operates on a population level with Pareto selection, while SIMBA focuses on individual hard examples.

GEPA vs TextGrad

TextGrad and GEPA both use LLM feedback to optimize prompts, but their architectures are different enough that they excel in different scenarios.

The fundamental difference: TextGrad refines a single prompt iteratively, like gradient descent on a single trajectory. GEPA maintains a population of diverse candidates and combines their strengths, like evolutionary search with Pareto selection. For tasks where different evaluation instances require different strategies (multi-hop QA, math with varied problem types, agentic workflows with branching logic), GEPA's population approach avoids the local optima that single-candidate methods converge to.

TextGrad is simpler to set up and can work well on tasks with uniform difficulty. GEPA is the better choice when your evaluation set contains heterogeneous problem types that benefit from specialized strategies.

Benchmark Results

The ICLR 2026 paper evaluated GEPA across six tasks against GRPO (a top RL method) and MIPROv2. All results are on Qwen3 8B unless noted otherwise.

MATH Benchmark

GEPA achieves 93% accuracy on the MATH benchmark using a DSPy ChainOfThought program, compared to 67% with the unoptimized ChainOfThought baseline. The 26-point improvement comes from instruction refinement alone, with no few-shot examples, no architectural changes, and no model fine-tuning.

AIME 2025

On AIME 2025 (competition-level math), GEPA with GPT-4.1 Mini achieves 10% gains over the unoptimized baseline. Against MIPROv2 on the same task, GEPA achieves +12% accuracy. Competition math problems are heterogeneous by design, which means different problems reward different reasoning strategies. GEPA's Pareto frontier preserves specialized strategies for geometry, algebra, and number theory rather than averaging them into a single "best" prompt.

Enterprise Structured Extraction

On a facility support analyzer task (extracting structured information from enterprise documents), GEPA achieves a metric of 2.93/3 (97.8%). On financial entity extraction, GEPA delivers +14 points over the DSPy baseline and +22 points over the raw OpenAI baseline.

Code Examples

Basic GEPA Usage in DSPy

The minimal setup requires a DSPy program, a metric function that returns ASI feedback, and a reflection LLM:

import dspy
from dspy import GEPA

# Configure your task LM
lm = dspy.LM("openai/gpt-5.4-mini")
dspy.configure(lm=lm)

# Define your program
class MathSolver(dspy.Module):
    def __init__(self):
        self.solve = dspy.ChainOfThought("question: str -> answer: str")

    def forward(self, question: str):
        return self.solve(question=question)

# Define a metric with feedback (ASI)
def math_metric(example, pred, trace=None):
    correct = example.answer.strip() == pred.answer.strip()
    # Return feedback as ASI for GEPA to reflect on
    if not correct:
        feedback = f"Expected '{example.answer}' but got '{pred.answer}'. "
        feedback += f"The reasoning was: {getattr(pred, 'reasoning', 'N/A')}"
        return correct, feedback
    return correct, "Correct answer."

# Initialize GEPA
optimizer = GEPA(
    metric=math_metric,
    auto="light",              # "light", "medium", or "heavy"
    num_threads=8,
    reflection_lm=dspy.LM(
        model="openai/gpt-5.4",  # Strong model for reflection
        temperature=1.0,
        max_tokens=16000
    )
)

# Compile
optimized = optimizer.compile(
    MathSolver(),
    trainset=train_examples,
    valset=val_examples,
)

# Save the optimized program
optimized.save("math_solver_gepa.json")

GEPA with Rich ASI Feedback

The quality of GEPA's optimization scales with the quality of ASI feedback. Here is an example with detailed diagnostic information:

def extraction_metric(example, pred, trace=None):
    """Metric that returns rich ASI for GEPA reflection."""
    score = 0.0
    feedback_parts = []

    # Check each expected field
    for field in example.expected_fields:
        predicted = getattr(pred, field, None)
        expected = getattr(example, field)

        if predicted == expected:
            score += 1.0
            feedback_parts.append(f"[CORRECT] {field}: '{predicted}'")
        elif predicted is None:
            score += 0.0
            feedback_parts.append(
                f"[MISSING] {field}: expected '{expected}', got None. "
                f"The model did not extract this field at all."
            )
        else:
            score += 0.0
            feedback_parts.append(
                f"[WRONG] {field}: expected '{expected}', got '{predicted}'. "
                f"Possible cause: the model may have confused this with "
                f"a similar field or extracted from the wrong section."
            )

    normalized = score / len(example.expected_fields)
    feedback = "\n".join(feedback_parts)
    return normalized, feedback

GEPA Configuration Options

optimizer = GEPA(
    metric=metric_with_feedback,

    # Preset: "light" (fast), "medium" (balanced), "heavy" (thorough)
    auto="medium",

    # Reflection model (use a strong LLM here)
    reflection_lm=dspy.LM(
        model="openai/gpt-5.4",
        temperature=1.0,
        max_tokens=32000
    ),

    # How many examples per reflection step (default: 2)
    # Smaller = more focused reflections, larger = broader view
    reflection_minibatch_size=3,

    # Candidate selection strategy
    # "pareto" = sample from Pareto frontier (default)
    candidate_selection_strategy="pareto",

    # Parallelism
    num_threads=16,

    # Track optimization statistics
    track_stats=True,
)

optimized = optimizer.compile(
    program,
    trainset=train_set,
    valset=val_set,  # If not provided, uses trainset for both
)

When to Use GEPA

The Decision Flow

GEPA Beyond DSPy: optimize_anything

GEPA is not limited to prompt optimization within DSPy. The optimize_anything API, available via pip install gepa, can optimize any text artifact against any evaluation function. You declare the artifact, the evaluator, and optional background knowledge. GEPA handles the rest.

from gepa import optimize_anything

# Optimize a Python function for performance
result = optimize_anything(
    artifact="""
def process_data(records):
    results = []
    for record in records:
        if record['status'] == 'active':
            results.append(transform(record))
    return results
""",
    evaluator=benchmark_function,  # returns (score, feedback)
    background="Python 3.12, target: minimize runtime on 100k records",
    mode="single_task",
    reflection_lm="openai/gpt-5.4",
)

print(result.optimized_artifact)  # Optimized code
print(result.score)               # Final score

The three optimization modes cover different use cases:

Integration with Coding Agents

GEPA-optimized prompts are only as useful as the infrastructure that executes them. In coding agent workflows, two bottlenecks dominate: finding the right code context and applying generated edits.

A GEPA-optimized coding agent might produce better instructions for code modification, but if the agent spends 60% of its time searching for the right files, the optimized prompt's improvements are diluted. Similarly, if code edits are applied slowly or inaccurately, the optimization loop's iteration speed drops.

The optimization feedback loop works like this: GEPA evolves the agent's system prompt. Each candidate prompt is evaluated by running the agent on a set of coding tasks. WarpGrep handles the context retrieval step within each evaluation. Morph Fast Apply handles the code edit application. The evaluator returns ASI describing what the agent did wrong (missed a file, applied the edit to the wrong location, failed to handle an edge case), and GEPA uses this to propose the next prompt variant.

Frequently Asked Questions

What is GEPA in DSPy?

GEPA (Genetic-Pareto) is a reflective prompt optimizer in DSPy that evolves instructions using natural language reflection on execution traces. Instead of collapsing feedback into a scalar reward, GEPA reads error messages, reasoning logs, and profiling data to diagnose why a prompt failed and propose targeted fixes. It was accepted as an ICLR 2026 oral and outperforms both MIPROv2 (+13% aggregate) and reinforcement learning methods like GRPO (+20% with 35x fewer rollouts).

How does GEPA differ from MIPROv2?

MIPROv2 uses Bayesian optimization to search over instruction and demonstration combinations, requiring 40+ trials and 200+ examples for best results. GEPA uses reflective prompt evolution with a Pareto frontier, requiring as few as 10 examples and 20-100 evaluations. GEPA focuses on instruction-only optimization without few-shot demonstrations, while MIPROv2 optimizes both. In head-to-head benchmarks, GEPA outperforms MIPROv2 by 13% aggregate across all tasks and models.

What is Actionable Side Information (ASI)?

ASI is diagnostic feedback returned by the evaluator alongside the score. It can include error messages, profiling traces, constraint violations, agent reasoning logs, or rendered images via VLMs. ASI serves as the text-optimization analogue of a gradient: it tells the LLM reflector why a candidate failed and how to fix it. The richer the ASI, the fewer evaluations GEPA needs to converge.

What is the GEPA Pareto frontier?

The Pareto frontier is the set of prompt candidates where each one achieves the highest score on at least one evaluation instance. A prompt survives on the frontier as long as no other single prompt beats it on every test case. This preserves diverse strategies that excel on different problem types, preventing the optimizer from converging to a single local optimum.

Can GEPA be used outside of DSPy?

Yes. GEPA ships as a standalone library (pip install gepa). The optimize_anything API can optimize any text artifact: prompts, code, agent architectures, configurations, and more. It supports three modes: single-task search, multi-task search with cross-task transfer, and generalization to unseen problems.

How many examples does GEPA need?

GEPA is highly sample-efficient. It can produce improvements with as few as 10 training examples and 20-100 total evaluations. RL methods like GRPO typically need 10,000+ rollouts. MIPROv2 recommends 200+ examples with 40+ trials. GEPA achieves this efficiency through reflective feedback and Pareto-aware candidate selection, which eliminates most of the random exploration that other methods require.

What benchmarks has GEPA been tested on?

The ICLR 2026 paper evaluated GEPA across six tasks including AIME 2025 (math), HotpotQA (multi-hop QA), PUPA, and HoVer. On Qwen3 8B, it outperformed GRPO by up to 20% and MIPROv2 by 13% aggregate. On the MATH benchmark, GEPA-optimized programs reached 93% accuracy vs 67% with basic ChainOfThought. On structured extraction tasks, GEPA achieved 97.8% quality scores.

What is the relationship between GEPA and SIMBA?

Both are instruction-level optimizers in DSPy. SIMBA uses stochastic mini-batch sampling to find examples with high output variability, then applies LLM introspection to generate improvement rules. GEPA operates on a population level with Pareto selection and reflective evolution. GEPA achieves higher quality but uses more compute per generation. SIMBA is better suited for targeted improvement on specific failure cases within an otherwise working system.

How does GEPA compare to TextGrad?

Both use LLM feedback, but their search strategies differ. TextGrad iteratively refines a single prompt using textual feedback from a critic LLM, like gradient descent on a single trajectory. GEPA maintains a population on a Pareto frontier and combines complementary strategies through evolutionary merge operations. GEPA's population-based approach avoids local optima that single-candidate methods like TextGrad can converge to, especially on heterogeneous evaluation sets.

What LLM should I use as the GEPA reflection model?

Use a strong model for reflection: GPT-5.4, Claude Sonnet 4, or better. The reflector analyzes execution traces, diagnoses failure modes, and proposes targeted prompt improvements. The task model (the "student") can be cheaper since it handles more frequent calls. Reflection calls are infrequent relative to task evaluations, so the cost of a stronger reflector is amortized.

Related Articles

• DSPy Prompt Optimization: Complete Guide to All Optimizers
• Prompt Compression: Reducing Token Usage Without Quality Loss
• Best AI Coding Agents in 2026
• AI Coding Benchmarks 2026