Scoring Deep Dive¶

This notebook explains how GEPA computes scores from Karenina's verification results. Understanding scoring is crucial for:

1. Interpreting optimization progress
2. Balancing correctness vs quality
3. Multi-model optimization with Pareto frontiers
4. Diagnosing template/rubric failures

---

Setup¶

In [ ]:

import sys
from pathlib import Path

sys.path.insert(0, str(Path.cwd().parent.parent.parent / "src"))

from karenina import Benchmark
from karenina.integrations.gepa import (
    ObjectiveConfig,
    TraitSelectionMode,
    compute_improvement,
    compute_objective_scores,
    extract_failed_fields,
)
from karenina.schemas import ModelConfig, VerificationConfig

# Load benchmark
benchmark_path = Path.home() / "Projects/karenina-monorepo/local_data/data/checkpoints/aime_2025.jsonld"
benchmark = Benchmark.load(benchmark_path)
print(f"Loaded: {benchmark.name} ({len(benchmark.get_question_ids())} questions)")

import sys from pathlib import Path sys.path.insert(0, str(Path.cwd().parent.parent.parent / "src")) from karenina import Benchmark from karenina.integrations.gepa import ( ObjectiveConfig, TraitSelectionMode, compute_improvement, compute_objective_scores, extract_failed_fields, ) from karenina.schemas import ModelConfig, VerificationConfig # Load benchmark benchmark_path = Path.home() / "Projects/karenina-monorepo/local_data/data/checkpoints/aime_2025.jsonld" benchmark = Benchmark.load(benchmark_path) print(f"Loaded: {benchmark.name} ({len(benchmark.get_question_ids())} questions)")

---

Run Verification to Get Results¶

First, let's run verification to get some real results to score.

In [ ]:

# Configure verification with Claude Haiku
config = VerificationConfig(
    answering_models=[
        ModelConfig(
            id="claude-haiku",
            model_provider="anthropic",
            model_name="claude-haiku-4-5",
            temperature=0.0,
            interface="langchain",
            system_prompt="You are a math expert. Solve the problem and provide the final integer answer (0-999).",
        )
    ],
    parsing_models=[
        ModelConfig(
            id="claude-haiku-parser",
            model_provider="anthropic",
            model_name="claude-haiku-4-5",
            temperature=0.0,
            interface="langchain",
        )
    ],
    evaluation_mode="template_only",
    replicate_count=1,
)

# Run on a small subset
question_ids = benchmark.get_question_ids()[:5]
print(f"Running verification on {len(question_ids)} questions...")

results = benchmark.run_verification(config, question_ids=question_ids)
print(f"Got {len(results.results)} results")

# Configure verification with Claude Haiku config = VerificationConfig( answering_models=[ ModelConfig( id="claude-haiku", model_provider="anthropic", model_name="claude-haiku-4-5", temperature=0.0, interface="langchain", system_prompt="You are a math expert. Solve the problem and provide the final integer answer (0-999).", ) ], parsing_models=[ ModelConfig( id="claude-haiku-parser", model_provider="anthropic", model_name="claude-haiku-4-5", temperature=0.0, interface="langchain", ) ], evaluation_mode="template_only", replicate_count=1, ) # Run on a small subset question_ids = benchmark.get_question_ids()[:5] print(f"Running verification on {len(question_ids)} questions...") results = benchmark.run_verification(config, question_ids=question_ids) print(f"Got {len(results.results)} results")

---

compute_objective_scores(): Scoring One Result¶

The core scoring function converts a VerificationResult into per-dimension objective scores. It returns a dict keyed by model:dimension (for example claude-haiku:template), each a float in 0.0-1.0. Which dimensions appear is controlled by an ObjectiveConfig: the template dimension (correctness) and one entry per rubric trait when traits are included.

In [ ]:

# Score individual results (template dimension only)
score_config = ObjectiveConfig(include_template=True, trait_mode=TraitSelectionMode.NONE)

for result in results.results:
    model_name = result.metadata.answering_model or "model"
    objectives = compute_objective_scores(result, model_name, score_config)
    # Mean across the returned objective dimensions
    score = sum(objectives.values()) / len(objectives) if objectives else 0.0

    passed = result.template.verify_result if result.template else False
    status = "PASS" if passed else "FAIL"

    print(f"{status} | Score: {score:.2f} | Question: {result.metadata.question_text[:50]}...")

# Score individual results (template dimension only) score_config = ObjectiveConfig(include_template=True, trait_mode=TraitSelectionMode.NONE) for result in results.results: model_name = result.metadata.answering_model or "model" objectives = compute_objective_scores(result, model_name, score_config) # Mean across the returned objective dimensions score = sum(objectives.values()) / len(objectives) if objectives else 0.0 passed = result.template.verify_result if result.template else False status = "PASS" if passed else "FAIL" print(f"{status} | Score: {score:.2f} | Question: {result.metadata.question_text[:50]}...")

Understanding the Objective Dimensions¶

compute_objective_scores produces one entry per objective dimension:

• template: 1.0 when verify_result is True, else 0.0.
• one entry per rubric trait: boolean traits map to 1.0/0.0, score traits normalize by their max_score, and metric traits expand into the metrics enabled in metric_config (for example f1).

Each entry is keyed as model:dimension, so the same trait on two models becomes two separate objectives. To collapse the dict into a single number, average its values (as shown above).

In [ ]:

# Score the same result under different objective configurations
result = results.results[0]
model_name = result.metadata.answering_model or "model"

# Template only (correctness)
template_only = compute_objective_scores(
    result, model_name, ObjectiveConfig(include_template=True, trait_mode=TraitSelectionMode.NONE)
)

# All rubric traits, no template
traits_only = compute_objective_scores(
    result, model_name, ObjectiveConfig(include_template=False, trait_mode=TraitSelectionMode.ALL)
)

# Template plus all traits
template_and_traits = compute_objective_scores(
    result, model_name, ObjectiveConfig(include_template=True, trait_mode=TraitSelectionMode.ALL)
)

print(f"Template-only objectives: {template_only}")
print(f"Traits-only objectives: {traits_only}")
print(f"Template + traits objectives: {template_and_traits}")

# Score the same result under different objective configurations result = results.results[0] model_name = result.metadata.answering_model or "model" # Template only (correctness) template_only = compute_objective_scores( result, model_name, ObjectiveConfig(include_template=True, trait_mode=TraitSelectionMode.NONE) ) # All rubric traits, no template traits_only = compute_objective_scores( result, model_name, ObjectiveConfig(include_template=False, trait_mode=TraitSelectionMode.ALL) ) # Template plus all traits template_and_traits = compute_objective_scores( result, model_name, ObjectiveConfig(include_template=True, trait_mode=TraitSelectionMode.ALL) ) print(f"Template-only objectives: {template_only}") print(f"Traits-only objectives: {traits_only}") print(f"Template + traits objectives: {template_and_traits}")

---

Aggregating Scores Across Results¶

To get a single benchmark-level number, average the per-result scores. There is no separate aggregation function: reduce each result with compute_objective_scores and take the mean.

In [ ]:

# Reduce each result to its mean objective score, then average across questions
score_config = ObjectiveConfig(include_template=True, trait_mode=TraitSelectionMode.NONE)


def mean_objective_score(result):
    model_name = result.metadata.answering_model or "model"
    objectives = compute_objective_scores(result, model_name, score_config)
    return sum(objectives.values()) / len(objectives) if objectives else 0.0


aggregate_score = sum(mean_objective_score(r) for r in results.results) / len(results.results)
print(f"Aggregate score across {len(results.results)} questions: {aggregate_score:.2%}")

# Reduce each result to its mean objective score, then average across questions score_config = ObjectiveConfig(include_template=True, trait_mode=TraitSelectionMode.NONE) def mean_objective_score(result): model_name = result.metadata.answering_model or "model" objectives = compute_objective_scores(result, model_name, score_config) return sum(objectives.values()) / len(objectives) if objectives else 0.0 aggregate_score = sum(mean_objective_score(r) for r in results.results) / len(results.results) print(f"Aggregate score across {len(results.results)} questions: {aggregate_score:.2%}")

In [ ]:

# Manual calculation to verify
individual_scores = [mean_objective_score(r) for r in results.results]
manual_avg = sum(individual_scores) / len(individual_scores)

print(f"Individual scores: {individual_scores}")
print(f"Manual average: {manual_avg:.2%}")
print(f"Aggregate: {aggregate_score:.2%}")
print(f"Match: {abs(manual_avg - aggregate_score) < 1e-6}")

# Manual calculation to verify individual_scores = [mean_objective_score(r) for r in results.results] manual_avg = sum(individual_scores) / len(individual_scores) print(f"Individual scores: {individual_scores}") print(f"Manual average: {manual_avg:.2%}") print(f"Aggregate: {aggregate_score:.2%}") print(f"Match: {abs(manual_avg - aggregate_score) < 1e-6}")

---

Multi-Model Scoring¶

For multi-model benchmarks, compute_objective_scores already namespaces each dimension by model. Passing each model's result with its own model_name yields keys like claude-haiku:template and claude-sonnet:template, which GEPA treats as separate Pareto objectives.

In [ ]:

# Run verification with two models
multi_model_config = VerificationConfig(
    answering_models=[
        ModelConfig(
            id="claude-haiku",
            model_provider="anthropic",
            model_name="claude-haiku-4-5",
            temperature=0.0,
            interface="langchain",
            system_prompt="You are a math expert. Give the final integer answer.",
        ),
        ModelConfig(
            id="claude-sonnet",
            model_provider="anthropic",
            model_name="claude-sonnet-4-5",
            temperature=0.0,
            interface="langchain",
            system_prompt="You are a math expert. Show your work and give the final integer answer.",
        ),
    ],
    parsing_models=[
        ModelConfig(
            id="parser",
            model_provider="anthropic",
            model_name="claude-haiku-4-5",
            temperature=0.0,
            interface="langchain",
        )
    ],
    evaluation_mode="template_only",
    replicate_count=1,
)

# Run on subset
print("Running multi-model verification on 3 questions...")
multi_results = benchmark.run_verification(multi_model_config, question_ids=question_ids[:3])
print(f"Got {len(multi_results.results)} results")

# Run verification with two models multi_model_config = VerificationConfig( answering_models=[ ModelConfig( id="claude-haiku", model_provider="anthropic", model_name="claude-haiku-4-5", temperature=0.0, interface="langchain", system_prompt="You are a math expert. Give the final integer answer.", ), ModelConfig( id="claude-sonnet", model_provider="anthropic", model_name="claude-sonnet-4-5", temperature=0.0, interface="langchain", system_prompt="You are a math expert. Show your work and give the final integer answer.", ), ], parsing_models=[ ModelConfig( id="parser", model_provider="anthropic", model_name="claude-haiku-4-5", temperature=0.0, interface="langchain", ) ], evaluation_mode="template_only", replicate_count=1, ) # Run on subset print("Running multi-model verification on 3 questions...") multi_results = benchmark.run_verification(multi_model_config, question_ids=question_ids[:3]) print(f"Got {len(multi_results.results)} results")

In [ ]:

# Group results by model
from collections import defaultdict

results_by_model = defaultdict(list)
for result in multi_results.results:
    model_name = result.metadata.answering_model or "unknown"
    results_by_model[model_name].append(result)

print("Results per model:")
for model, model_results in results_by_model.items():
    print(f"  {model}: {len(model_results)} results")

# Group results by model from collections import defaultdict results_by_model = defaultdict(list) for result in multi_results.results: model_name = result.metadata.answering_model or "unknown" results_by_model[model_name].append(result) print("Results per model:") for model, model_results in results_by_model.items(): print(f" {model}: {len(model_results)} results")

In [ ]:

# Compute per-model objective scores; keys are namespaced as model:dimension
score_config = ObjectiveConfig(include_template=True, trait_mode=TraitSelectionMode.NONE)

model_scores = {}
for model_name, model_results in results_by_model.items():
    per_result = []
    for result in model_results:
        objectives = compute_objective_scores(result, model_name, score_config)
        per_result.append(sum(objectives.values()) / len(objectives) if objectives else 0.0)
    model_scores[model_name] = sum(per_result) / len(per_result) if per_result else 0.0

overall_score = sum(model_scores.values()) / len(model_scores) if model_scores else 0.0

print(f"Overall score: {overall_score:.2%}")
print("\nPer-model scores:")
for model, score in model_scores.items():
    print(f"  {model}: {score:.2%}")

# Compute per-model objective scores; keys are namespaced as model:dimension score_config = ObjectiveConfig(include_template=True, trait_mode=TraitSelectionMode.NONE) model_scores = {} for model_name, model_results in results_by_model.items(): per_result = [] for result in model_results: objectives = compute_objective_scores(result, model_name, score_config) per_result.append(sum(objectives.values()) / len(objectives) if objectives else 0.0) model_scores[model_name] = sum(per_result) / len(per_result) if per_result else 0.0 overall_score = sum(model_scores.values()) / len(model_scores) if model_scores else 0.0 print(f"Overall score: {overall_score:.2%}") print("\nPer-model scores:") for model, score in model_scores.items(): print(f" {model}: {score:.2%}")

---

extract_failed_fields(): Diagnosing Failures¶

Identify which template fields failed verification.

In [ ]:

# Find a failed result
failed_results = [r for r in results.results if r.template and not r.template.verify_result]

if failed_results:
    failed = failed_results[0]

    # Extract failed fields
    failed_fields = extract_failed_fields(failed)

    print(f"Question: {failed.metadata.question_text[:60]}...")
    print(f"Expected answer: {failed.metadata.raw_answer}")
    print(f"Model response: {failed.template.raw_llm_response[:100]}...")
    print(f"\nFailed fields: {failed_fields}")
else:
    print("All results passed! No failures to analyze.")
    print("\nExample of extract_failed_fields() with a passing result:")
    print(f"Failed fields: {extract_failed_fields(results.results[0])}")

# Find a failed result failed_results = [r for r in results.results if r.template and not r.template.verify_result] if failed_results: failed = failed_results[0] # Extract failed fields failed_fields = extract_failed_fields(failed) print(f"Question: {failed.metadata.question_text[:60]}...") print(f"Expected answer: {failed.metadata.raw_answer}") print(f"Model response: {failed.template.raw_llm_response[:100]}...") print(f"\nFailed fields: {failed_fields}") else: print("All results passed! No failures to analyze.") print("\nExample of extract_failed_fields() with a passing result:") print(f"Failed fields: {extract_failed_fields(results.results[0])}")

---

compute_improvement(): Measuring Optimization Progress¶

Compute relative improvement from baseline to optimized score.

In [ ]:

# Example: Baseline vs optimized scores
baseline_score = 0.60  # 60% before optimization
optimized_score = 0.75  # 75% after optimization

improvement = compute_improvement(baseline_score, optimized_score)

print(f"Baseline: {baseline_score:.2%}")
print(f"Optimized: {optimized_score:.2%}")
print(f"Improvement: {improvement:.2%} ({improvement * 100:.1f}% relative improvement)")

# Example: Baseline vs optimized scores baseline_score = 0.60 # 60% before optimization optimized_score = 0.75 # 75% after optimization improvement = compute_improvement(baseline_score, optimized_score) print(f"Baseline: {baseline_score:.2%}") print(f"Optimized: {optimized_score:.2%}") print(f"Improvement: {improvement:.2%} ({improvement * 100:.1f}% relative improvement)")

In [ ]:

# Edge cases
print("Edge cases:")

# Improvement from zero baseline
imp_from_zero = compute_improvement(0.0, 0.50)
print(f"  From 0% to 50%: {imp_from_zero:.2%} (returns absolute score when baseline is 0)")

# Negative improvement (worse)
negative_imp = compute_improvement(0.70, 0.60)
print(f"  From 70% to 60%: {negative_imp:.2%} (negative = got worse)")

# No change
no_change = compute_improvement(0.50, 0.50)
print(f"  No change: {no_change:.2%}")

# Edge cases print("Edge cases:") # Improvement from zero baseline imp_from_zero = compute_improvement(0.0, 0.50) print(f" From 0% to 50%: {imp_from_zero:.2%} (returns absolute score when baseline is 0)") # Negative improvement (worse) negative_imp = compute_improvement(0.70, 0.60) print(f" From 70% to 60%: {negative_imp:.2%} (negative = got worse)") # No change no_change = compute_improvement(0.50, 0.50) print(f" No change: {no_change:.2%}")

---

Practical Scoring Examples¶

Example 1: AIME Benchmark (Correctness Only)¶

In [ ]:

# For AIME, we only care about correctness (template dimension)
score_config = ObjectiveConfig(include_template=True, trait_mode=TraitSelectionMode.NONE)

aime_scores = []
for result in results.results:
    model_name = result.metadata.answering_model or "model"
    objectives = compute_objective_scores(result, model_name, score_config)
    aime_scores.append(sum(objectives.values()) / len(objectives) if objectives else 0.0)

accuracy = sum(aime_scores) / len(aime_scores)
print(f"AIME Accuracy: {accuracy:.2%} ({sum(aime_scores):.0f}/{len(aime_scores)} correct)")

# For AIME, we only care about correctness (template dimension) score_config = ObjectiveConfig(include_template=True, trait_mode=TraitSelectionMode.NONE) aime_scores = [] for result in results.results: model_name = result.metadata.answering_model or "model" objectives = compute_objective_scores(result, model_name, score_config) aime_scores.append(sum(objectives.values()) / len(objectives) if objectives else 0.0) accuracy = sum(aime_scores) / len(aime_scores) print(f"AIME Accuracy: {accuracy:.2%} ({sum(aime_scores):.0f}/{len(aime_scores)} correct)")

Example 2: Tracking Optimization Progress¶

In [ ]:

# Simulate optimization progress
generation_scores = [
    0.40,  # Gen 0: Seed prompt
    0.45,  # Gen 1
    0.52,  # Gen 2
    0.58,  # Gen 3
    0.55,  # Gen 4 (regression)
    0.62,  # Gen 5
    0.65,  # Gen 6
    0.70,  # Gen 7: Best
]

baseline = generation_scores[0]
best_score = max(generation_scores)
best_gen = generation_scores.index(best_score)

print("Optimization Progress:")
for gen, score in enumerate(generation_scores):
    imp = compute_improvement(baseline, score)
    marker = " <- BEST" if gen == best_gen else ""
    print(f"  Gen {gen}: {score:.2%} (improvement: {imp:+.1%}){marker}")

final_improvement = compute_improvement(baseline, best_score)
print(f"\nTotal improvement: {final_improvement:+.1%} (from {baseline:.2%} to {best_score:.2%})")

# Simulate optimization progress generation_scores = [ 0.40, # Gen 0: Seed prompt 0.45, # Gen 1 0.52, # Gen 2 0.58, # Gen 3 0.55, # Gen 4 (regression) 0.62, # Gen 5 0.65, # Gen 6 0.70, # Gen 7: Best ] baseline = generation_scores[0] best_score = max(generation_scores) best_gen = generation_scores.index(best_score) print("Optimization Progress:") for gen, score in enumerate(generation_scores): imp = compute_improvement(baseline, score) marker = " <- BEST" if gen == best_gen else "" print(f" Gen {gen}: {score:.2%} (improvement: {imp:+.1%}){marker}") final_improvement = compute_improvement(baseline, best_score) print(f"\nTotal improvement: {final_improvement:+.1%} (from {baseline:.2%} to {best_score:.2%})")

Example 3: Multi-Model Pareto Analysis¶

In [ ]:

# Simulate multi-model scores for different prompts
candidates = {
    "Prompt A": {"haiku": 0.65, "sonnet": 0.70},
    "Prompt B": {"haiku": 0.70, "sonnet": 0.60},  # Better for haiku
    "Prompt C": {"haiku": 0.60, "sonnet": 0.75},  # Better for sonnet
    "Prompt D": {"haiku": 0.68, "sonnet": 0.72},  # Balanced
}

print("Multi-Model Candidate Analysis:")
print("-" * 50)

for prompt, scores in candidates.items():
    avg = sum(scores.values()) / len(scores)
    print(f"{prompt}:")
    for model, score in scores.items():
        print(f"  {model}: {score:.2%}")
    print(f"  Average: {avg:.2%}")
    print()

# Find Pareto-optimal candidates
print("Pareto Analysis:")
print("  - Prompt A: Not Pareto-optimal (dominated by D)")
print("  - Prompt B: Pareto-optimal for haiku")
print("  - Prompt C: Pareto-optimal for sonnet")
print("  - Prompt D: Pareto-optimal (best average)")

# Simulate multi-model scores for different prompts candidates = { "Prompt A": {"haiku": 0.65, "sonnet": 0.70}, "Prompt B": {"haiku": 0.70, "sonnet": 0.60}, # Better for haiku "Prompt C": {"haiku": 0.60, "sonnet": 0.75}, # Better for sonnet "Prompt D": {"haiku": 0.68, "sonnet": 0.72}, # Balanced } print("Multi-Model Candidate Analysis:") print("-" * 50) for prompt, scores in candidates.items(): avg = sum(scores.values()) / len(scores) print(f"{prompt}:") for model, score in scores.items(): print(f" {model}: {score:.2%}") print(f" Average: {avg:.2%}") print() # Find Pareto-optimal candidates print("Pareto Analysis:") print(" - Prompt A: Not Pareto-optimal (dominated by D)") print(" - Prompt B: Pareto-optimal for haiku") print(" - Prompt C: Pareto-optimal for sonnet") print(" - Prompt D: Pareto-optimal (best average)")

---

Summary¶

Function	Purpose	Returns
compute_objective_scores()	Per-dimension scores for one result	dict[str, float] keyed model:dimension
compute_improvement()	Relative improvement	float (fraction)
extract_failed_fields()	Find failed template fields	list[str]

Key Takeaways¶

1. Objective scores are per dimension, keyed as model:dimension in 0.0-1.0
2. Include only the template dimension for factual benchmarks like AIME
3. Per-model namespacing enables Pareto optimization across models
4. extract_failed_fields() helps diagnose verification failures
5. compute_improvement() tracks optimization progress

Next Steps¶

• 05_karenina_adapter.ipynb - Using the KareninaAdapter