Multi-Step Agent Evaluation¶

This tutorial shows how to evaluate agent workflows with distinct phases using TaskEval's step-scoped evaluation. Each step gets its own logs, templates, and rubric traits, evaluated independently. Use this when your agent performs sequential phases (retrieval, reasoning, synthesis) and you want to assess each phase separately: did retrieval find relevant sources? Did synthesis accurately summarize them?

What you'll learn:

• Log outputs to named steps with step_id
• Control log routing with target ("global", "step", "both")
• Attach step-specific templates and rubrics
• Log structured agent traces with tool calls using log_trace()
• Evaluate individual steps with evaluate(config, step_id=...)
• Evaluate globally (all steps combined)
• Compare per-step results via TaskEvalResult.per_step
• Use traces_only merge strategy to filter debug output

---

Complete Example¶

A compact multi-step evaluation end-to-end:

In [2]:

from karenina.benchmark.task_eval import TaskEval
from karenina.schemas.config.models import ModelConfig
from karenina.schemas.entities.rubric import LLMRubricTrait, Rubric
from karenina.schemas.verification.config import VerificationConfig

task = TaskEval(task_id="multi-step-demo")

task.log("Found 3 relevant papers on venetoclax.", step_id="retrieval")
task.log("BCL2 is the direct target of venetoclax.", step_id="synthesis")

task.add_rubric(
    Rubric(
        llm_traits=[
            LLMRubricTrait(
                name="source_quality",
                kind="boolean",
                higher_is_better=True,
                description="True if relevant sources were identified.",
            ),
        ]
    ),
    step_id="retrieval",
)

task.add_rubric(
    Rubric(
        llm_traits=[
            LLMRubricTrait(
                name="accuracy",
                kind="boolean",
                higher_is_better=True,
                description="True if the synthesis is factually accurate.",
            ),
        ]
    ),
    step_id="synthesis",
)

config = VerificationConfig(
    parsing_models=[
        ModelConfig(
            id="haiku",
            model_name="claude-haiku-4-5",
            model_provider="anthropic",
            interface="langchain",
            temperature=0.0,
        )
    ],
    parsing_only=True,
)

for sid in ["retrieval", "synthesis"]:
    step_result = task.evaluate(config, step_id=sid)
    stats = step_result.per_step[sid].get_summary_stats()
    print(f"Step '{sid}': {stats['rubric_traits_passed']}/{stats['rubric_traits_total']} traits passed")

from karenina.benchmark.task_eval import TaskEval from karenina.schemas.config.models import ModelConfig from karenina.schemas.entities.rubric import LLMRubricTrait, Rubric from karenina.schemas.verification.config import VerificationConfig task = TaskEval(task_id="multi-step-demo") task.log("Found 3 relevant papers on venetoclax.", step_id="retrieval") task.log("BCL2 is the direct target of venetoclax.", step_id="synthesis") task.add_rubric( Rubric( llm_traits=[ LLMRubricTrait( name="source_quality", kind="boolean", higher_is_better=True, description="True if relevant sources were identified.", ), ] ), step_id="retrieval", ) task.add_rubric( Rubric( llm_traits=[ LLMRubricTrait( name="accuracy", kind="boolean", higher_is_better=True, description="True if the synthesis is factually accurate.", ), ] ), step_id="synthesis", ) config = VerificationConfig( parsing_models=[ ModelConfig( id="haiku", model_name="claude-haiku-4-5", model_provider="anthropic", interface="langchain", temperature=0.0, ) ], parsing_only=True, ) for sid in ["retrieval", "synthesis"]: step_result = task.evaluate(config, step_id=sid) stats = step_result.per_step[sid].get_summary_stats() print(f"Step '{sid}': {stats['rubric_traits_passed']}/{stats['rubric_traits_total']} traits passed")

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_2', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_4', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_6', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_8', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Abstention detection failed with non-retryable error: Structured output failed after 4 attempts. Last error: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_10', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_12', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_14', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_16', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Batch evaluation failed: Structured output failed after 4 attempts. Last error: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Rubric evaluator initialization/configuration failed for question b09934ba6e17c4f51f43a8ad7f4faffb: Failed to evaluate rubric traits using batch strategy: Structured output failed after 4 attempts. Last error: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_18', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_20', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_22', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_24', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Abstention detection failed with non-retryable error: Structured output failed after 4 attempts. Last error: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_26', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_28', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_30', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_32', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Batch evaluation failed: Structured output failed after 4 attempts. Last error: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Rubric evaluator initialization/configuration failed for question c2a195f75adcf3e4c69280d908b1c7b4: Failed to evaluate rubric traits using batch strategy: Structured output failed after 4 attempts. Last error: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Step 'retrieval': 0/0 traits passed
Step 'synthesis': 0/0 traits passed

---

Set Up the Agent Workflow¶

This tutorial uses a three-phase drug target analysis scenario:

1. Retrieval: Search databases for relevant sources
2. Reasoning: Analyze retrieved information and identify the target
3. Synthesis: Produce a final answer with supporting evidence

Each phase generates different output that should be evaluated against different criteria.

In [3]:

from karenina.benchmark.task_eval import TaskEval

task = TaskEval(task_id="drug-target-agent")

print(f"Created TaskEval: {task.task_id}")
print("Phases: retrieval → reasoning → synthesis")

from karenina.benchmark.task_eval import TaskEval task = TaskEval(task_id="drug-target-agent") print(f"Created TaskEval: {task.task_id}") print("Phases: retrieval → reasoning → synthesis")

Created TaskEval: drug-target-agent
Phases: retrieval → reasoning → synthesis

---

Log Step Outputs¶

Pass step_id to log() to assign output to a named step. The target parameter controls where the log is stored:

Target	Step scope	Global scope	Default when
"both"	Yes	Yes	step_id is set
"step"	Yes	No	Never (must specify explicitly)
"global"	No	Yes	step_id is not set

In [4]:

# Retrieval phase outputs
task.log(
    "Database search returned 3 results for 'venetoclax mechanism of action': "
    "(1) Souers et al. 2013, Nat Med; (2) Roberts et al. 2016, NEJM; "
    "(3) Anderson et al. 2016, Blood.",
    step_id="retrieval",
)

# Reasoning phase outputs
task.log(
    "Analysis: All three sources identify BCL2 as the direct binding target. "
    "Souers et al. describe the BH3 mimetic mechanism. Roberts et al. confirm "
    "clinical efficacy via BCL2 inhibition in CLL patients.",
    step_id="reasoning",
)

# Synthesis phase outputs
task.log(
    "The approved pharmacological target of venetoclax is BCL2 (B-cell lymphoma 2). "
    "Venetoclax is a selective BH3 mimetic that binds BCL2, displacing pro-apoptotic "
    "proteins and triggering apoptosis [Souers 2013, Roberts 2016].",
    step_id="synthesis",
)

print(f"Global logs: {len(task.global_logs)} (all steps contribute with target='both')")

# Retrieval phase outputs task.log( "Database search returned 3 results for 'venetoclax mechanism of action': " "(1) Souers et al. 2013, Nat Med; (2) Roberts et al. 2016, NEJM; " "(3) Anderson et al. 2016, Blood.", step_id="retrieval", ) # Reasoning phase outputs task.log( "Analysis: All three sources identify BCL2 as the direct binding target. " "Souers et al. describe the BH3 mimetic mechanism. Roberts et al. confirm " "clinical efficacy via BCL2 inhibition in CLL patients.", step_id="reasoning", ) # Synthesis phase outputs task.log( "The approved pharmacological target of venetoclax is BCL2 (B-cell lymphoma 2). " "Venetoclax is a selective BH3 mimetic that binds BCL2, displacing pro-apoptotic " "proteins and triggering apoptosis [Souers 2013, Roberts 2016].", step_id="synthesis", ) print(f"Global logs: {len(task.global_logs)} (all steps contribute with target='both')")

Global logs: 3 (all steps contribute with target='both')

Step-Only Logging¶

Use target="step" when a log should not appear in global evaluation. This is useful for intermediate debug output that is relevant to one step but should not influence the overall assessment.

In [5]:

task.log(
    "DEBUG: search latency was 1.2s, 3/50 results passed relevance filter",
    step_id="retrieval",
    target="step",
    level="debug",
)

print(f"Global logs unchanged: {len(task.global_logs)}")

task.log( "DEBUG: search latency was 1.2s, 3/50 results passed relevance filter", step_id="retrieval", target="step", level="debug", ) print(f"Global logs unchanged: {len(task.global_logs)}")

Global logs unchanged: 3

---

Log Structured Traces¶

Use log_trace() when you need to preserve conversation structure, tool calls, and speaker roles. This matters for rubric traits that assess tool usage or multi-turn reasoning.

In [6]:

from karenina.ports.messages import Message, ToolUseContent

task.log_trace(
    [
        Message.user("What is the approved pharmacological target of venetoclax?"),
        Message.assistant(
            "Let me search the drug database.",
            tool_calls=[ToolUseContent(id="call_1", name="search_drugs", input={"query": "venetoclax target"})],
        ),
        Message.tool_result(tool_use_id="call_1", content="BCL2 (B-cell lymphoma 2); selective BH3 mimetic"),
        Message.assistant(
            "Based on the database search, the approved pharmacological target of "
            "venetoclax is BCL2 (B-cell lymphoma 2). Venetoclax acts as a selective "
            "BH3 mimetic that directly binds to the BCL2 protein."
        ),
    ],
    step_id="retrieval",
)

print(f"Global logs after trace: {len(task.global_logs)}")

from karenina.ports.messages import Message, ToolUseContent task.log_trace( [ Message.user("What is the approved pharmacological target of venetoclax?"), Message.assistant( "Let me search the drug database.", tool_calls=[ToolUseContent(id="call_1", name="search_drugs", input={"query": "venetoclax target"})], ), Message.tool_result(tool_use_id="call_1", content="BCL2 (B-cell lymphoma 2); selective BH3 mimetic"), Message.assistant( "Based on the database search, the approved pharmacological target of " "venetoclax is BCL2 (B-cell lymphoma 2). Venetoclax acts as a selective " "BH3 mimetic that directly binds to the BCL2 protein." ), ], step_id="retrieval", ) print(f"Global logs after trace: {len(task.global_logs)}")

Global logs after trace: 4

---

Attach Step-Scoped Criteria¶

Pass step_id to add_rubric() to attach criteria to a specific step. Each step can have different traits, tuned to what that phase should accomplish.

In [7]:

from karenina.schemas.entities.rubric import LLMRubricTrait, RegexRubricTrait, Rubric

# Retrieval: did it find relevant sources?
task.add_rubric(
    Rubric(
        llm_traits=[
            LLMRubricTrait(
                name="source_relevance",
                kind="boolean",
                higher_is_better=True,
                description=(
                    "True if the retrieved sources are directly relevant to the query about "
                    "venetoclax's pharmacological target. False if sources are tangential or unrelated."
                ),
            ),
            LLMRubricTrait(
                name="source_count",
                kind="score",
                min_score=1,
                max_score=5,
                higher_is_better=True,
                description=(
                    "Rate the number and diversity of sources retrieved. "
                    "1: no sources or only one. 3: adequate coverage. 5: comprehensive multi-source retrieval."
                ),
            ),
        ],
    ),
    step_id="retrieval",
)

print("Added retrieval rubric (2 traits)")

from karenina.schemas.entities.rubric import LLMRubricTrait, RegexRubricTrait, Rubric # Retrieval: did it find relevant sources? task.add_rubric( Rubric( llm_traits=[ LLMRubricTrait( name="source_relevance", kind="boolean", higher_is_better=True, description=( "True if the retrieved sources are directly relevant to the query about " "venetoclax's pharmacological target. False if sources are tangential or unrelated." ), ), LLMRubricTrait( name="source_count", kind="score", min_score=1, max_score=5, higher_is_better=True, description=( "Rate the number and diversity of sources retrieved. " "1: no sources or only one. 3: adequate coverage. 5: comprehensive multi-source retrieval." ), ), ], ), step_id="retrieval", ) print("Added retrieval rubric (2 traits)")

Added retrieval rubric (2 traits)

In [8]:

# Reasoning: is the analysis sound?
task.add_rubric(
    Rubric(
        llm_traits=[
            LLMRubricTrait(
                name="reasoning_quality",
                kind="boolean",
                higher_is_better=True,
                description=(
                    "True if the reasoning step correctly synthesizes information from "
                    "retrieved sources. False if it introduces unsupported claims or "
                    "ignores contradictory evidence."
                ),
            ),
        ],
    ),
    step_id="reasoning",
)

print("Added reasoning rubric (1 trait)")

# Reasoning: is the analysis sound? task.add_rubric( Rubric( llm_traits=[ LLMRubricTrait( name="reasoning_quality", kind="boolean", higher_is_better=True, description=( "True if the reasoning step correctly synthesizes information from " "retrieved sources. False if it introduces unsupported claims or " "ignores contradictory evidence." ), ), ], ), step_id="reasoning", ) print("Added reasoning rubric (1 trait)")

Added reasoning rubric (1 trait)

In [9]:

# Synthesis: is the final answer accurate and well-supported?
task.add_rubric(
    Rubric(
        llm_traits=[
            LLMRubricTrait(
                name="factual_accuracy",
                kind="boolean",
                higher_is_better=True,
                description="True if the final answer is factually correct.",
            ),
        ],
        regex_traits=[
            RegexRubricTrait(
                name="has_citations",
                pattern=r"\[.+?\d{4}\]|\(.+?\d{4}\)",
                description="Response includes author-year citations.",
                higher_is_better=True,
            ),
        ],
    ),
    step_id="synthesis",
)

print("Added synthesis rubric (1 LLM + 1 regex trait)")

# Synthesis: is the final answer accurate and well-supported? task.add_rubric( Rubric( llm_traits=[ LLMRubricTrait( name="factual_accuracy", kind="boolean", higher_is_better=True, description="True if the final answer is factually correct.", ), ], regex_traits=[ RegexRubricTrait( name="has_citations", pattern=r"\[.+?\d{4}\]|\(.+?\d{4}\)", description="Response includes author-year citations.", higher_is_better=True, ), ], ), step_id="synthesis", ) print("Added synthesis rubric (1 LLM + 1 regex trait)")

Added synthesis rubric (1 LLM + 1 regex trait)

---

Attach a Global Template¶

A global template evaluates the combined output from all steps. This checks whether the overall workflow produced the correct answer, regardless of how individual steps performed.

In [10]:

from pydantic import Field
from karenina.schemas.entities import BaseAnswer


class Answer(BaseAnswer):
    identifies_bcl2: bool = Field(
        description=(
            "True if the combined output identifies BCL2 (including Bcl-2, BCL-2, or "
            "B-cell lymphoma 2) as the direct pharmacological target of venetoclax."
        )
    )

    def ground_truth(self):
        self.correct = {"identifies_bcl2": True}

    def verify(self) -> bool:
        return self.identifies_bcl2 == self.correct["identifies_bcl2"]


task.add_template(Answer)

print("Added global template for correctness verification")

from pydantic import Field from karenina.schemas.entities import BaseAnswer class Answer(BaseAnswer): identifies_bcl2: bool = Field( description=( "True if the combined output identifies BCL2 (including Bcl-2, BCL-2, or " "B-cell lymphoma 2) as the direct pharmacological target of venetoclax." ) ) def ground_truth(self): self.correct = {"identifies_bcl2": True} def verify(self) -> bool: return self.identifies_bcl2 == self.correct["identifies_bcl2"] task.add_template(Answer) print("Added global template for correctness verification")

Added global template for correctness verification

---

Evaluate Per Step¶

Evaluate each step individually by passing step_id. Each step evaluation uses only that step's logs and criteria.

In [11]:

from karenina.schemas.config.models import ModelConfig
from karenina.schemas.verification.config import VerificationConfig

config = VerificationConfig(
    parsing_models=[
        ModelConfig(
            id="claude-haiku-4-5",
            model_name="claude-haiku-4-5",
            model_provider="anthropic",
            interface="langchain",
            temperature=0.0,
        )
    ],
    parsing_only=True,
)

for step_id in ["retrieval", "reasoning", "synthesis"]:
    step_result = task.evaluate(config, step_id=step_id)
    step_eval = step_result.per_step[step_id]
    stats = step_eval.get_summary_stats()
    print(f"Step '{step_id}': {stats['rubric_traits_passed']}/{stats['rubric_traits_total']} rubric traits passed")

from karenina.schemas.config.models import ModelConfig from karenina.schemas.verification.config import VerificationConfig config = VerificationConfig( parsing_models=[ ModelConfig( id="claude-haiku-4-5", model_name="claude-haiku-4-5", model_provider="anthropic", interface="langchain", temperature=0.0, ) ], parsing_only=True, ) for step_id in ["retrieval", "reasoning", "synthesis"]: step_result = task.evaluate(config, step_id=step_id) step_eval = step_result.per_step[step_id] stats = step_eval.get_summary_stats() print(f"Step '{step_id}': {stats['rubric_traits_passed']}/{stats['rubric_traits_total']} rubric traits passed")

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_34', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_36', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_38', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_40', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Abstention detection failed with non-retryable error: Structured output failed after 4 attempts. Last error: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_42', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_44', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_46', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_48', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Batch evaluation failed: Structured output failed after 4 attempts. Last error: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Rubric evaluator initialization/configuration failed for question b09934ba6e17c4f51f43a8ad7f4faffb: Failed to evaluate rubric traits using batch strategy: Structured output failed after 4 attempts. Last error: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_50', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_52', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_54', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_56', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Abstention detection failed with non-retryable error: Structured output failed after 4 attempts. Last error: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_58', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_60', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_62', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_64', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Batch evaluation failed: Structured output failed after 4 attempts. Last error: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Rubric evaluator initialization/configuration failed for question 909cc00384f8a2f38c66fe04b5e405c4: Failed to evaluate rubric traits using batch strategy: Structured output failed after 4 attempts. Last error: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_66', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_68', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_70', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_72', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Abstention detection failed with non-retryable error: Structured output failed after 4 attempts. Last error: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_74', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_76', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_78', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_80', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Batch evaluation failed: Structured output failed after 4 attempts. Last error: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Rubric evaluator initialization/configuration failed for question c2a195f75adcf3e4c69280d908b1c7b4: Failed to evaluate rubric traits using batch strategy: Structured output failed after 4 attempts. Last error: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Step 'retrieval': 0/0 rubric traits passed
Step 'reasoning': 0/0 rubric traits passed
Step 'synthesis': 0/0 rubric traits passed

---

Evaluate Globally¶

Call evaluate() without step_id to run global evaluation. Global evaluation uses all globally-scoped logs (from steps with target="both" or target="global") and globally-attached criteria (the template in this case).

In [12]:

global_result = task.evaluate(config)

if global_result.global_eval:
    stats = global_result.global_eval.get_summary_stats()
    print(f"Global: {stats['traces_passed']}/{stats['traces_total']} passed template verification")
    print(f"Global: {stats['rubric_traits_passed']}/{stats['rubric_traits_total']} rubric traits passed")

global_result = task.evaluate(config) if global_result.global_eval: stats = global_result.global_eval.get_summary_stats() print(f"Global: {stats['traces_passed']}/{stats['traces_total']} passed template verification") print(f"Global: {stats['rubric_traits_passed']}/{stats['rubric_traits_total']} rubric traits passed")

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_82', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_84', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_86', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_88', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Abstention detection failed with non-retryable error: Structured output failed after 4 attempts. Last error: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AnswerWithID
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_90', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AnswerWithID
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_92', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AnswerWithID
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_94', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AnswerWithID
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_96', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Stage ParseTemplate set error: Parsing failed: Failed to parse response into Answer after all retry strategies: [{'type': 'tool_use', 'id': 'fixture_97', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Stage FinalizeResult set error: Parsing failed: Failed to parse response into Answer after all retry strategies: [{'type': 'tool_use', 'id': 'fixture_97', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_99', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_101', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_103', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_105', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Abstention detection failed with non-retryable error: Structured output failed after 4 attempts. Last error: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_107', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_109', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_111', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_113', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Batch evaluation failed: Structured output failed after 4 attempts. Last error: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Rubric evaluator initialization/configuration failed for question b09934ba6e17c4f51f43a8ad7f4faffb: Failed to evaluate rubric traits using batch strategy: Structured output failed after 4 attempts. Last error: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_115', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_117', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_119', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_121', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Abstention detection failed with non-retryable error: Structured output failed after 4 attempts. Last error: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_123', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_125', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_127', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_129', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Batch evaluation failed: Structured output failed after 4 attempts. Last error: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Rubric evaluator initialization/configuration failed for question 909cc00384f8a2f38c66fe04b5e405c4: Failed to evaluate rubric traits using batch strategy: Structured output failed after 4 attempts. Last error: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_131', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_133', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_135', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_137', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Abstention detection failed with non-retryable error: Structured output failed after 4 attempts. Last error: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_139', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_141', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_143', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_145', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Batch evaluation failed: Structured output failed after 4 attempts. Last error: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Rubric evaluator initialization/configuration failed for question c2a195f75adcf3e4c69280d908b1c7b4: Failed to evaluate rubric traits using batch strategy: Structured output failed after 4 attempts. Last error: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Global: 0/1 passed template verification
Global: 0/0 rubric traits passed

---

Inspect Per-Step Results¶

The TaskEvalResult.per_step dict maps step IDs to StepEval objects. Each StepEval contains verification_results and provides get_summary_stats().

In [13]:

print(f"Steps evaluated: {list(global_result.per_step.keys())}")

for step_id, step_eval in global_result.per_step.items():
    stats = step_eval.get_summary_stats()
    print(f"\n--- {step_id} ---")
    print(f"  Traces: {stats['traces_total']}")
    print(f"  Rubric traits: {stats['rubric_traits_passed']}/{stats['rubric_traits_total']}")

    for qid, vr_list in step_eval.verification_results.items():
        vr = vr_list[0]
        if vr.rubric:
            for name, score in vr.rubric.llm_trait_scores.items():
                print(f"  {name}: {score}")
            for name, score in vr.rubric.regex_trait_scores.items():
                print(f"  {name}: {score}")

print(f"Steps evaluated: {list(global_result.per_step.keys())}") for step_id, step_eval in global_result.per_step.items(): stats = step_eval.get_summary_stats() print(f"\n--- {step_id} ---") print(f" Traces: {stats['traces_total']}") print(f" Rubric traits: {stats['rubric_traits_passed']}/{stats['rubric_traits_total']}") for qid, vr_list in step_eval.verification_results.items(): vr = vr_list[0] if vr.rubric: for name, score in vr.rubric.llm_trait_scores.items(): print(f" {name}: {score}") for name, score in vr.rubric.regex_trait_scores.items(): print(f" {name}: {score}")

Steps evaluated: ['retrieval', 'reasoning', 'synthesis']

--- retrieval ---
  Traces: 1
  Rubric traits: 0/0

--- reasoning ---
  Traces: 1
  Rubric traits: 0/0

--- synthesis ---
  Traces: 1
  Rubric traits: 0/0

---

Merge Strategies in Multi-Step¶

The traces_only merge strategy is particularly useful in multi-step workflows. When steps produce both debug output (via log()) and structured traces (via log_trace()), traces_only ensures only the structured content reaches the judge.

In [14]:

filtered_task = TaskEval(task_id="filtered-demo", merge_strategy="traces_only")

# Debug output: excluded from evaluation
filtered_task.log("DEBUG: starting retrieval, timeout=30s", step_id="retrieval")
filtered_task.log("DEBUG: 3 results returned in 1.2s", step_id="retrieval")

# Structured trace: included in evaluation
filtered_task.log_trace(
    [
        Message.assistant("Found 3 papers on venetoclax: Souers 2013, Roberts 2016, Anderson 2016."),
    ],
    step_id="retrieval",
)

print(f"Total logs: {len(filtered_task.global_logs)}")
print("With traces_only: only the Message trace reaches the judge")
print("Both debug log() calls are silently excluded")

filtered_task = TaskEval(task_id="filtered-demo", merge_strategy="traces_only") # Debug output: excluded from evaluation filtered_task.log("DEBUG: starting retrieval, timeout=30s", step_id="retrieval") filtered_task.log("DEBUG: 3 results returned in 1.2s", step_id="retrieval") # Structured trace: included in evaluation filtered_task.log_trace( [ Message.assistant("Found 3 papers on venetoclax: Souers 2013, Roberts 2016, Anderson 2016."), ], step_id="retrieval", ) print(f"Total logs: {len(filtered_task.global_logs)}") print("With traces_only: only the Message trace reaches the judge") print("Both debug log() calls are silently excluded")

Total logs: 3
With traces_only: only the Message trace reaches the judge
Both debug log() calls are silently excluded

You can also override the merge strategy per evaluate() call:

In [15]:

# Override just for one evaluation
# result = task.evaluate(config, merge_strategy="traces_only")
print("Per-call override: task.evaluate(config, merge_strategy='traces_only')")

# Override just for one evaluation # result = task.evaluate(config, merge_strategy="traces_only") print("Per-call override: task.evaluate(config, merge_strategy='traces_only')")

Per-call override: task.evaluate(config, merge_strategy='traces_only')

---

Cleanup¶

---

Next Steps¶

• Basic Evaluation: Single-step evaluation with templates and rubrics
• Quality Assessment: Rubric-only evaluation with all trait types
• TaskEval concept page: Object structure, pipeline integration, merge strategies
• Rubrics: All five trait types in depth
• Verification Pipeline: The 13-stage engine (with sub-stages 7a/7b and 11a/11b) that TaskEval feeds into