Analyzing Verification Results with DataFrames¶

This guide covers how to analyze verification results using the DataFrame-first approach, providing flexible and powerful data analysis with pandas.

Overview¶

What This Guide Covers¶

This guide focuses on analyzing verification results after you've run verification. The DataFrame-first approach provides a modern, flexible way to wrangle and analyze verification output using pandas DataFrames.

Typical Workflow:

1. Run verification (see verification.md for details)
2. Extract results using result_set.get_template_results(), get_rubrics_results(), or get_judgment_results()
3. Convert to DataFrame for analysis
4. Analyze with pandas

Quick Start¶

Basic Workflow¶

After running verification, you can analyze results with DataFrames:

In [38]:

# STEP 1: Extract result type wrappers from verification output
template_results = result_set.get_template_results()
rubric_results = result_set.get_rubrics_results()
judgment_results = result_set.get_judgment_results()

print(f"Template results: {len(template_results)}")
print(f"Rubric results: {len(rubric_results)}")
print(f"Judgment results: {len(judgment_results)}")

# STEP 1: Extract result type wrappers from verification output template_results = result_set.get_template_results() rubric_results = result_set.get_rubrics_results() judgment_results = result_set.get_judgment_results() print(f"Template results: {len(template_results)}") print(f"Rubric results: {len(rubric_results)}") print(f"Judgment results: {len(judgment_results)}")

Template results: 10
Rubric results: 10
Judgment results: 10

In [39]:

# STEP 2: Convert verification results to DataFrames
template_df = template_results.to_dataframe()
rubric_df = rubric_results.to_dataframe()
judgment_df = judgment_results.to_dataframe()

print(f"Template DataFrame shape: {template_df.shape}")
print(f"Rubric DataFrame shape: {rubric_df.shape}")
print(f"Judgment DataFrame shape: {judgment_df.shape}")

# STEP 2: Convert verification results to DataFrames template_df = template_results.to_dataframe() rubric_df = rubric_results.to_dataframe() judgment_df = judgment_results.to_dataframe() print(f"Template DataFrame shape: {template_df.shape}") print(f"Rubric DataFrame shape: {rubric_df.shape}") print(f"Judgment DataFrame shape: {judgment_df.shape}")

Template DataFrame shape: (18, 34)
Rubric DataFrame shape: (30, 17)
Judgment DataFrame shape: (10, 40)

In [40]:

# STEP 3: Analyze with pandas
pass_rate = template_df.groupby("question_id")["field_match"].mean()
print("Pass rate by question:")
print(pass_rate)

# STEP 3: Analyze with pandas pass_rate = template_df.groupby("question_id")["field_match"].mean() print("Pass rate by question:") print(pass_rate)

Pass rate by question:
question_id
q_chromosomes          1.0
q_drug_target          1.0
q_hemoglobin           1.0
q_inflammatory_lung    1.0
q_partial_fail         0.5
Name: field_match, dtype: float64

Basic Example: Template Verification Analysis¶

In [41]:

# Get template results DataFrame
df = template_results.to_dataframe()

# View available columns
print("Available columns:")
print(df.columns.tolist())

# Get template results DataFrame df = template_results.to_dataframe() # View available columns print("Available columns:") print(df.columns.tolist())

Available columns:
['success', 'failure_category', 'failure_group', 'failure_stage', 'failure_reason', 'caveats', 'recursion_limit_reached', 'question_id', 'template_id', 'question_text', 'keywords', 'replicate', 'answering_mcp_servers', 'answering_model', 'parsing_model', 'answering_system_prompt', 'parsing_system_prompt', 'raw_llm_response', 'field_name', 'gt_value', 'llm_value', 'field_match', 'field_type', 'verify_result', 'embedding_check_performed', 'embedding_similarity_score', 'embedding_model_used', 'embedding_override_applied', 'abstention_check_performed', 'abstention_detected', 'abstention_reasoning', 'abstention_override_applied', 'regex_validations_performed', 'regex_overall_success', 'execution_time', 'timestamp', 'run_name', 'result_index']

In [42]:

# Filter to successful verifications only
successful = df[df["success"] == True]
print(f"Successful verifications: {len(successful)} out of {len(df)}")

# Filter to successful verifications only successful = df[df["success"] == True] print(f"Successful verifications: {len(successful)} out of {len(df)}")

Successful verifications: 18 out of 18

In [43]:

# Calculate pass rate by question
pass_rates = successful.groupby("question_id")["field_match"].mean()
print("\nPass rates by question:")
print(pass_rates)

# Calculate pass rate by question pass_rates = successful.groupby("question_id")["field_match"].mean() print("\nPass rates by question:") print(pass_rates)

Pass rates by question:
question_id
q_chromosomes          1.0
q_drug_target          1.0
q_hemoglobin           1.0
q_inflammatory_lung    1.0
q_partial_fail         0.5
Name: field_match, dtype: float64

In [44]:

# Find questions with low pass rates
low_performers = pass_rates[pass_rates < 1.0]
print(f"\nQuestions with <100% pass rate: {len(low_performers)}")
print(low_performers)

# Find questions with low pass rates low_performers = pass_rates[pass_rates < 1.0] print(f"\nQuestions with <100% pass rate: {len(low_performers)}") print(low_performers)

Questions with <100% pass rate: 1
question_id
q_partial_fail    0.5
Name: field_match, dtype: float64

In [45]:

# Analyze by field
field_performance = successful.groupby("field_name").agg({"field_match": ["mean", "count"]})
print("\nField performance:")
print(field_performance)

# Analyze by field field_performance = successful.groupby("field_name").agg({"field_match": ["mean", "count"]}) print("\nField performance:") print(field_performance)

Field performance:
            field_match      
                   mean count
field_name                   
class               1.0     2
composition         1.0     2
count               1.0     2
diseases            1.0     2
subunits            1.0     2
target              1.0     2
unit                1.0     4
weight              0.0     2

DataFrame Methods Reference¶

TemplateResults¶

to_dataframe()¶

Convert template verification results to pandas DataFrame with field-level explosion.

Key Columns:

• Status: success, failure_category, failure_group, failure_stage, failure_reason, caveats, recursion_limit_reached
• Field Comparison: field_name, gt_value, llm_value, field_match, field_type
• Verification Checks: embedding_check_performed, embedding_similarity_score, abstention_detected, regex_validations_performed
• Identification: question_id, template_id, answering_model, parsing_model

In [46]:

df = template_results.to_dataframe()

# Each field gets its own row
# For example, a question with 3 fields → 3 rows
print(f"Total rows (exploded by field): {len(df)}")
print(f"Unique questions: {df['question_id'].nunique()}")

df = template_results.to_dataframe() # Each field gets its own row # For example, a question with 3 fields → 3 rows print(f"Total rows (exploded by field): {len(df)}") print(f"Unique questions: {df['question_id'].nunique()}")

Total rows (exploded by field): 18
Unique questions: 5

In [47]:

# Analyze specific fields
weight_fields = df[df["field_name"] == "weight"]
if len(weight_fields) > 0:
    match_rate = weight_fields["field_match"].mean()
    print(f"Weight field match rate: {match_rate:.2%}")

# Analyze specific fields weight_fields = df[df["field_name"] == "weight"] if len(weight_fields) > 0: match_rate = weight_fields["field_match"].mean() print(f"Weight field match rate: {match_rate:.2%}")

Weight field match rate: 0.00%

to_regex_dataframe()¶

Convert regex validation results to DataFrame with pattern explosion.

In [48]:

# For regex validation results
regex_df = template_results.to_regex_dataframe()
print(f"Regex DataFrame rows: {len(regex_df)}")

if len(regex_df) > 0:
    # Analyze pattern success rates
    pattern_stats = regex_df.groupby("pattern_name").agg({"matched": "mean"})
    print("\nPattern success rates:")
    print(pattern_stats)
else:
    print("No regex validations performed in this example.")

# For regex validation results regex_df = template_results.to_regex_dataframe() print(f"Regex DataFrame rows: {len(regex_df)}") if len(regex_df) > 0: # Analyze pattern success rates pattern_stats = regex_df.groupby("pattern_name").agg({"matched": "mean"}) print("\nPattern success rates:") print(pattern_stats) else: print("No regex validations performed in this example.")

Regex DataFrame rows: 0
No regex validations performed in this example.

to_usage_dataframe(totals_only=False)¶

Convert token usage data to DataFrame.

In [49]:

# Detailed usage by stage
usage_df = template_results.to_usage_dataframe(totals_only=False)
stage_costs = usage_df.groupby("usage_stage")["total_tokens"].sum()
print("Token usage by stage:")
print(stage_costs)

# Detailed usage by stage usage_df = template_results.to_usage_dataframe(totals_only=False) stage_costs = usage_df.groupby("usage_stage")["total_tokens"].sum() print("Token usage by stage:") print(stage_costs)

Token usage by stage:
usage_stage
answer_generation    500
parsing              300
Name: total_tokens, dtype: int64

In [50]:

# Total usage only
totals_df = template_results.to_usage_dataframe(totals_only=True)
total_cost = totals_df["total_tokens"].sum()
print(f"\nTotal tokens used: {total_cost}")

# Total usage only totals_df = template_results.to_usage_dataframe(totals_only=True) total_cost = totals_df["total_tokens"].sum() print(f"\nTotal tokens used: {total_cost}")

Total tokens used: 800

RubricResults¶

to_dataframe(trait_type="all")¶

Convert rubric evaluation results to DataFrame with trait explosion.

Parameters:

• trait_type: Filter trait type - "llm_score", "llm_binary", "llm", "regex", "callable", "metric", or "all"

Key Columns:

• trait_name: Name of the rubric trait
• trait_type: Type (llm_score, llm_binary, regex, callable, metric)
• trait_score: Score value

In [51]:

# Get all LLM-scored traits
llm_df = rubric_results.to_dataframe(trait_type="llm_score")
print(f"LLM trait DataFrame shape: {llm_df.shape}")
print("\nSample LLM trait scores:")
print(llm_df[["question_id", "trait_name", "trait_score"]].head(10))

# Get all LLM-scored traits llm_df = rubric_results.to_dataframe(trait_type="llm_score") print(f"LLM trait DataFrame shape: {llm_df.shape}") print("\nSample LLM trait scores:") print(llm_df[["question_id", "trait_name", "trait_score"]].head(10))

LLM trait DataFrame shape: (30, 17)

Sample LLM trait scores:
           question_id    trait_name  trait_score
0        q_chromosomes      Accuracy            5
1        q_chromosomes       Clarity            4
2        q_chromosomes  Completeness            4
3        q_drug_target      Accuracy            5
4        q_drug_target       Clarity            5
5        q_drug_target  Completeness            5
6         q_hemoglobin      Accuracy            5
7         q_hemoglobin       Clarity            4
8         q_hemoglobin  Completeness            5
9  q_inflammatory_lung      Accuracy            4

In [52]:

# Analyze trait performance
trait_scores = llm_df.groupby("trait_name")["trait_score"].agg(["mean", "std", "count"])
print("\nTrait score summary:")
print(trait_scores)

# Analyze trait performance trait_scores = llm_df.groupby("trait_name")["trait_score"].agg(["mean", "std", "count"]) print("\nTrait score summary:") print(trait_scores)

Trait score summary:
              mean       std  count
trait_name                         
Accuracy       4.4  0.843274     10
Clarity        4.6  0.516398     10
Completeness   4.4  0.516398     10

In [53]:

# Find low-scoring traits
low_scores = trait_scores[trait_scores["mean"] < 4.0]
print("\nTraits with average score < 4.0:")
print(low_scores)

# Find low-scoring traits low_scores = trait_scores[trait_scores["mean"] < 4.0] print("\nTraits with average score < 4.0:") print(low_scores)

Traits with average score < 4.0:
Empty DataFrame
Columns: [mean, std, count]
Index: []

In [54]:

# Compare models
model_comparison = llm_df.pivot_table(
    values="trait_score", index="question_id", columns="answering_model", aggfunc="mean"
)
print("\nModel comparison by question:")
print(model_comparison)

# Compare models model_comparison = llm_df.pivot_table( values="trait_score", index="question_id", columns="answering_model", aggfunc="mean" ) print("\nModel comparison by question:") print(model_comparison)

Model comparison by question:
answering_model      claude-3-5-sonnet  gpt-4o-mini
question_id                                        
q_chromosomes                 4.333333     4.333333
q_drug_target                 5.000000     5.000000
q_hemoglobin                  4.666667     4.666667
q_inflammatory_lung           4.333333     4.333333
q_partial_fail                4.000000     4.000000

Common Patterns¶

Pattern 1: Calculate Pass Rates¶

Template verification pass rate by question:

In [55]:

df = template_results.to_dataframe()

# Filter to successful verifications
successful = df[df["success"] == True]

# Calculate pass rate by question
pass_rates = successful.groupby("question_id")["field_match"].mean()
print("Pass rates by question:")
print(pass_rates)

# Get questions below threshold
failing = pass_rates[pass_rates < 1.0]
print(f"\nQuestions with <100% pass rate: {list(failing.index)}")

df = template_results.to_dataframe() # Filter to successful verifications successful = df[df["success"] == True] # Calculate pass rate by question pass_rates = successful.groupby("question_id")["field_match"].mean() print("Pass rates by question:") print(pass_rates) # Get questions below threshold failing = pass_rates[pass_rates < 1.0] print(f"\nQuestions with <100% pass rate: {list(failing.index)}")

Pass rates by question:
question_id
q_chromosomes          1.0
q_drug_target          1.0
q_hemoglobin           1.0
q_inflammatory_lung    1.0
q_partial_fail         0.5
Name: field_match, dtype: float64

Questions with <100% pass rate: ['q_partial_fail']

Rubric trait scores by model:

In [56]:

rubric_df = rubric_results.to_dataframe(trait_type="llm_score")

# Average score by model
model_scores = rubric_df.groupby("answering_model")["trait_score"].mean()
print("Average trait scores by model:")
print(model_scores)

rubric_df = rubric_results.to_dataframe(trait_type="llm_score") # Average score by model model_scores = rubric_df.groupby("answering_model")["trait_score"].mean() print("Average trait scores by model:") print(model_scores)

Average trait scores by model:
answering_model
claude-3-5-sonnet    4.466667
gpt-4o-mini          4.466667
Name: trait_score, dtype: float64

In [57]:

# Detailed breakdown by trait and model
model_trait_scores = rubric_df.pivot_table(
    values="trait_score", index="answering_model", columns="trait_name", aggfunc="mean"
)
print("\nModel scores by trait:")
print(model_trait_scores)

# Detailed breakdown by trait and model model_trait_scores = rubric_df.pivot_table( values="trait_score", index="answering_model", columns="trait_name", aggfunc="mean" ) print("\nModel scores by trait:") print(model_trait_scores)

Model scores by trait:
trait_name         Accuracy  Clarity  Completeness
answering_model                                   
claude-3-5-sonnet       4.4      4.6           4.4
gpt-4o-mini             4.4      4.6           4.4

Pattern 2: Multi-Dimensional Analysis¶

Template + Rubric combined analysis:

In [58]:

# Get DataFrames
template_df = template_results.to_dataframe()
rubric_df = rubric_results.to_dataframe(trait_type="llm_score")

# Aggregate to question level
template_agg = template_df.groupby("question_id")["field_match"].mean()
rubric_agg = rubric_df.groupby("question_id")["trait_score"].mean()

# Merge
combined = pd.DataFrame({"template_pass_rate": template_agg, "rubric_avg_score": rubric_agg})

print("Combined template and rubric metrics:")
print(combined)

# Get DataFrames template_df = template_results.to_dataframe() rubric_df = rubric_results.to_dataframe(trait_type="llm_score") # Aggregate to question level template_agg = template_df.groupby("question_id")["field_match"].mean() rubric_agg = rubric_df.groupby("question_id")["trait_score"].mean() # Merge combined = pd.DataFrame({"template_pass_rate": template_agg, "rubric_avg_score": rubric_agg}) print("Combined template and rubric metrics:") print(combined)

Combined template and rubric metrics:
                     template_pass_rate  rubric_avg_score
question_id                                              
q_chromosomes                       1.0          4.333333
q_drug_target                       1.0          5.000000
q_hemoglobin                        1.0          4.666667
q_inflammatory_lung                 1.0          4.333333
q_partial_fail                      0.5          4.000000

Pattern 3: Field-Level Analysis¶

Identify problematic fields:

In [59]:

df = template_results.to_dataframe()
successful = df[df["success"] == True]

# Calculate match rate by field
field_performance = successful.groupby("field_name").agg({"field_match": ["mean", "count", "sum"]})

# Sort by match rate
field_performance = field_performance.sort_values(("field_match", "mean"))

print("Field performance (worst to best):")
print(field_performance)

df = template_results.to_dataframe() successful = df[df["success"] == True] # Calculate match rate by field field_performance = successful.groupby("field_name").agg({"field_match": ["mean", "count", "sum"]}) # Sort by match rate field_performance = field_performance.sort_values(("field_match", "mean")) print("Field performance (worst to best):") print(field_performance)

Field performance (worst to best):
            field_match          
                   mean count sum
field_name                       
weight              0.0     2   0
class               1.0     2   2
composition         1.0     2   2
count               1.0     2   2
diseases            1.0     2   2
subunits            1.0     2   2
target              1.0     2   2
unit                1.0     4   4

Pattern 4: Model Comparison¶

In [60]:

df = template_results.to_dataframe()

# Pivot: questions × models
model_comparison = df.pivot_table(values="field_match", index="question_id", columns="answering_model", aggfunc="mean")

print("Model comparison by question:")
print(model_comparison)

df = template_results.to_dataframe() # Pivot: questions × models model_comparison = df.pivot_table(values="field_match", index="question_id", columns="answering_model", aggfunc="mean") print("Model comparison by question:") print(model_comparison)

Model comparison by question:
answering_model      claude-3-5-sonnet  gpt-4o-mini
question_id                                        
q_chromosomes                      1.0          1.0
q_drug_target                      1.0          1.0
q_hemoglobin                       1.0          1.0
q_inflammatory_lung                1.0          1.0
q_partial_fail                     0.5          0.5

In [61]:

# Calculate relative performance
# First compute the spread before adding string columns
model_comparison = model_comparison.copy()
model_comparison["performance_spread"] = model_comparison.max(axis=1) - model_comparison.min(axis=1)
model_comparison["best_model"] = model_comparison.idxmax(axis=1)

print("\nWith best model and spread:")
print(model_comparison)

# Calculate relative performance # First compute the spread before adding string columns model_comparison = model_comparison.copy() model_comparison["performance_spread"] = model_comparison.max(axis=1) - model_comparison.min(axis=1) model_comparison["best_model"] = model_comparison.idxmax(axis=1) print("\nWith best model and spread:") print(model_comparison)

With best model and spread:
answering_model      claude-3-5-sonnet  gpt-4o-mini  performance_spread  \
question_id                                                               
q_chromosomes                      1.0          1.0                 0.0   
q_drug_target                      1.0          1.0                 0.0   
q_hemoglobin                       1.0          1.0                 0.0   
q_inflammatory_lung                1.0          1.0                 0.0   
q_partial_fail                     0.5          0.5                 0.0   

answering_model             best_model  
question_id                             
q_chromosomes        claude-3-5-sonnet  
q_drug_target        claude-3-5-sonnet  
q_hemoglobin         claude-3-5-sonnet  
q_inflammatory_lung  claude-3-5-sonnet  
q_partial_fail       claude-3-5-sonnet  

In [62]:

# Find questions with high model variance
high_variance = model_comparison[model_comparison["performance_spread"] > 0]
print(f"\nQuestions with model variance: {len(high_variance)}")
print(high_variance[["best_model", "performance_spread"]])

# Find questions with high model variance high_variance = model_comparison[model_comparison["performance_spread"] > 0] print(f"\nQuestions with model variance: {len(high_variance)}") print(high_variance[["best_model", "performance_spread"]])

Questions with model variance: 0
Empty DataFrame
Columns: [best_model, performance_spread]
Index: []

Pattern 5: Error Analysis¶

In [63]:

df = template_results.to_dataframe()

# Get failed verifications (where field_match is False)
failed_fields = df[df["field_match"] == False]

if len(failed_fields) > 0:
    print("Failed field comparisons:")
    print(failed_fields[["question_id", "field_name", "gt_value", "llm_value"]].to_string())
else:
    print("No failed field comparisons in this dataset.")

df = template_results.to_dataframe() # Get failed verifications (where field_match is False) failed_fields = df[df["field_match"] == False] if len(failed_fields) > 0: print("Failed field comparisons:") print(failed_fields[["question_id", "field_name", "gt_value", "llm_value"]].to_string()) else: print("No failed field comparisons in this dataset.")

Failed field comparisons:
       question_id field_name gt_value llm_value
8   q_partial_fail     weight     5808      5800
17  q_partial_fail     weight     5808      5800

Pattern 7: Cost Analysis¶

In [64]:

usage_df = template_results.to_usage_dataframe(totals_only=True)

# Assuming cost per 1K tokens
INPUT_COST_PER_1K = 0.0001
OUTPUT_COST_PER_1K = 0.0003

usage_df["input_cost"] = usage_df["input_tokens"] / 1000 * INPUT_COST_PER_1K
usage_df["output_cost"] = usage_df["output_tokens"] / 1000 * OUTPUT_COST_PER_1K
usage_df["total_cost"] = usage_df["input_cost"] + usage_df["output_cost"]

# Cost by model
model_costs = usage_df.groupby("answering_model")["total_cost"].sum()
print("Cost by model:")
for model, cost in model_costs.items():
    print(f"  {model}: ${cost:.4f}")

# Total cost
print(f"\nTotal cost: ${usage_df['total_cost'].sum():.4f}")

usage_df = template_results.to_usage_dataframe(totals_only=True) # Assuming cost per 1K tokens INPUT_COST_PER_1K = 0.0001 OUTPUT_COST_PER_1K = 0.0003 usage_df["input_cost"] = usage_df["input_tokens"] / 1000 * INPUT_COST_PER_1K usage_df["output_cost"] = usage_df["output_tokens"] / 1000 * OUTPUT_COST_PER_1K usage_df["total_cost"] = usage_df["input_cost"] + usage_df["output_cost"] # Cost by model model_costs = usage_df.groupby("answering_model")["total_cost"].sum() print("Cost by model:") for model, cost in model_costs.items(): print(f" {model}: ${cost:.4f}") # Total cost print(f"\nTotal cost: ${usage_df['total_cost'].sum():.4f}")

Cost by model:
  claude-3-5-sonnet: $0.0001
  gpt-4o-mini: $0.0001

Total cost: $0.0001

Helper Methods¶

Helper methods provide convenient aggregations for common operations. They are implemented using the DataFrame API internally.

TemplateResults Helper Methods¶

aggregate_pass_rate(by="question_id", strategy="mean")¶

In [65]:

# Pass rate by question
question_rates = template_results.aggregate_pass_rate(by="question_id")
print("Pass rates by question:")
for qid, rate in question_rates.items():
    print(f"  {qid}: {rate:.2%}")

# Pass rate by question question_rates = template_results.aggregate_pass_rate(by="question_id") print("Pass rates by question:") for qid, rate in question_rates.items(): print(f" {qid}: {rate:.2%}")

Pass rates by question:
  q_chromosomes: 100.00%
  q_drug_target: 100.00%
  q_hemoglobin: 100.00%
  q_inflammatory_lung: 100.00%
  q_partial_fail: 50.00%

In [66]:

# Pass rate by model
model_rates = template_results.aggregate_pass_rate(by="answering_model")
print("\nPass rates by model:")
for model, rate in model_rates.items():
    print(f"  {model}: {rate:.2%}")

# Pass rate by model model_rates = template_results.aggregate_pass_rate(by="answering_model") print("\nPass rates by model:") for model, rate in model_rates.items(): print(f" {model}: {rate:.2%}")

Pass rates by model:
  claude-3-5-sonnet: 100.00%
  gpt-4o-mini: 80.00%

aggregate_embedding_scores(by="question_id", strategy="mean")¶

In [67]:

# Average embedding score by question
embedding_scores = template_results.aggregate_embedding_scores(by="question_id")
print("Embedding similarity scores by question:")
for qid, score in embedding_scores.items():
    print(f"  {qid}: {score:.3f}")

# Average embedding score by question embedding_scores = template_results.aggregate_embedding_scores(by="question_id") print("Embedding similarity scores by question:") for qid, score in embedding_scores.items(): print(f" {qid}: {score:.3f}")

Embedding similarity scores by question:
  q_chromosomes: 0.925
  q_drug_target: 0.895
  q_hemoglobin: 0.855
  q_inflammatory_lung: 0.825
  q_partial_fail: 0.875

RubricResults Helper Methods¶

aggregate_llm_traits(by="question_id", strategy="mean")¶

In [68]:

# Get average scores by question
question_traits = rubric_results.aggregate_llm_traits(by="question_id")

print("LLM trait scores by question:")
for qid, traits in question_traits.items():
    print(f"\n{qid}:")
    for trait, score in traits.items():
        print(f"  {trait}: {score:.2f}")

# Get average scores by question question_traits = rubric_results.aggregate_llm_traits(by="question_id") print("LLM trait scores by question:") for qid, traits in question_traits.items(): print(f"\n{qid}:") for trait, score in traits.items(): print(f" {trait}: {score:.2f}")

LLM trait scores by question:

q_chromosomes:
  Accuracy: 5.00
  Clarity: 4.00
  Completeness: 4.00

q_drug_target:
  Accuracy: 5.00
  Clarity: 5.00
  Completeness: 5.00

q_hemoglobin:
  Accuracy: 5.00
  Clarity: 4.00
  Completeness: 5.00

q_inflammatory_lung:
  Accuracy: 4.00
  Clarity: 5.00
  Completeness: 4.00

q_partial_fail:
  Accuracy: 3.00
  Clarity: 5.00
  Completeness: 4.00

In [69]:

# Get average scores by model
model_traits = rubric_results.aggregate_llm_traits(by="answering_model")

print("\nLLM trait scores by model:")
for model, traits in model_traits.items():
    print(f"\n{model}:")
    for trait, score in traits.items():
        print(f"  {trait}: {score:.2f}")

# Get average scores by model model_traits = rubric_results.aggregate_llm_traits(by="answering_model") print("\nLLM trait scores by model:") for model, traits in model_traits.items(): print(f"\n{model}:") for trait, score in traits.items(): print(f" {trait}: {score:.2f}")

LLM trait scores by model:

claude-3-5-sonnet:
  Accuracy: 4.40
  Clarity: 4.60
  Completeness: 4.40

gpt-4o-mini:
  Accuracy: 4.40
  Clarity: 4.60
  Completeness: 4.40

Performance Tips¶

1. Filter Early¶

Filter DataFrames before aggregation to reduce computational load:

In [70]:

# Good: Filter first
df = template_results.to_dataframe()
successful = df[df["success"] == True]
pass_rates = successful.groupby("question_id")["field_match"].mean()
print("Pass rates (filtered first):")
print(pass_rates)

# Good: Filter first df = template_results.to_dataframe() successful = df[df["success"] == True] pass_rates = successful.groupby("question_id")["field_match"].mean() print("Pass rates (filtered first):") print(pass_rates)

Pass rates (filtered first):
question_id
q_chromosomes          1.0
q_drug_target          1.0
q_hemoglobin           1.0
q_inflammatory_lung    1.0
q_partial_fail         0.5
Name: field_match, dtype: float64

2. Use Helper Methods for Simple Cases¶

In [71]:

# Prefer helper for simple aggregation
pass_rates = template_results.aggregate_pass_rate(by="question_id")
print("Pass rates using helper method:")
print(pass_rates)

# Prefer helper for simple aggregation pass_rates = template_results.aggregate_pass_rate(by="question_id") print("Pass rates using helper method:") print(pass_rates)

Pass rates using helper method:
{'q_chromosomes': 1.0, 'q_drug_target': 1.0, 'q_hemoglobin': 1.0, 'q_inflammatory_lung': 1.0, 'q_partial_fail': 0.5}

3. Avoid Repeated Conversions¶

In [72]:

# Good: Convert once
template_df = template_results.to_dataframe()
pass_rates = template_df.groupby("question_id")["field_match"].mean()
field_stats = template_df.groupby("field_name")["field_match"].agg(["mean", "count"])

print("Field statistics:")
print(field_stats)

# Good: Convert once template_df = template_results.to_dataframe() pass_rates = template_df.groupby("question_id")["field_match"].mean() field_stats = template_df.groupby("field_name")["field_match"].agg(["mean", "count"]) print("Field statistics:") print(field_stats)

Field statistics:
             mean  count
field_name              
class         1.0      2
composition   1.0      2
count         1.0      2
diseases      1.0      2
subunits      1.0      2
target        1.0      2
unit          1.0      4
weight        0.0      2

Summary¶

The DataFrame-first approach provides:

• Familiar API: Use standard pandas operations
• Flexibility: Combine multiple verification aspects in custom ways
• Performance: Leverage pandas' optimized operations
• Helper Methods: Convenience methods for common operations

For more information:

• See the DataFrame Quick Reference
• Check integration tests for real-world usage examples