Quality Assessment with Rubrics¶

This tutorial shows how to evaluate response quality using rubrics without an answer template. When there is no single correct answer (open-ended questions, creative tasks, safety assessments), rubric-only evaluation scores quality dimensions independently. Each trait uses a different evaluation strategy: LLM judgment, regex pattern matching, or custom Python functions.

What you'll learn:

• Set up rubric-only evaluation (no template needed)
• Define LLM traits: boolean and ordinal score kinds
• Define regex traits for pattern detection
• Define callable traits for custom Python checks
• Override serialized callables with callable_registry
• Combine multiple trait types in a single rubric
• Inspect rubric scores by trait type
• Compare quality across multiple logged outputs

---

Complete Example¶

Rubric-only evaluation in minimal form:

In [2]:

from karenina.benchmark.task_eval import TaskEval
from karenina.schemas.config.models import ModelConfig
from karenina.schemas.entities.rubric import LLMRubricTrait, Rubric
from karenina.schemas.verification.config import VerificationConfig

task = TaskEval(task_id="quality-check")
task.log("BCL2 is the primary target of venetoclax, a selective inhibitor used in CLL treatment [1].")

rubric = Rubric(
    llm_traits=[
        LLMRubricTrait(
            name="conciseness",
            kind="boolean",
            higher_is_better=True,
            description="True if the response is concise and directly answers the question.",
        ),
    ]
)
task.add_rubric(rubric)

config = VerificationConfig(
    parsing_models=[
        ModelConfig(
            id="haiku",
            model_name="claude-haiku-4-5",
            model_provider="anthropic",
            interface="langchain",
            temperature=0.0,
        )
    ],
    parsing_only=True,
)
result = task.evaluate(config)
print(result.summary())

from karenina.benchmark.task_eval import TaskEval from karenina.schemas.config.models import ModelConfig from karenina.schemas.entities.rubric import LLMRubricTrait, Rubric from karenina.schemas.verification.config import VerificationConfig task = TaskEval(task_id="quality-check") task.log("BCL2 is the primary target of venetoclax, a selective inhibitor used in CLL treatment [1].") rubric = Rubric( llm_traits=[ LLMRubricTrait( name="conciseness", kind="boolean", higher_is_better=True, description="True if the response is concise and directly answers the question.", ), ] ) task.add_rubric(rubric) config = VerificationConfig( parsing_models=[ ModelConfig( id="haiku", model_name="claude-haiku-4-5", model_provider="anthropic", interface="langchain", temperature=0.0, ) ], parsing_only=True, ) result = task.evaluate(config) print(result.summary())

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_2', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_4', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_6', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_8', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Abstention detection failed with non-retryable error: Structured output failed after 4 attempts. Last error: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_10', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_12', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_14', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_16', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Batch evaluation failed: Structured output failed after 4 attempts. Last error: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Rubric evaluator initialization/configuration failed for question 39430513e323dbe155b2201a8b877bd2: Failed to evaluate rubric traits using batch strategy: Structured output failed after 4 attempts. Last error: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

0/1 traces passed

When you attach only a rubric (no template), TaskEval automatically uses rubric_only evaluation mode. A synthetic minimal template is created internally so the pipeline can run.

---

Log the Agent Output¶

Start with a detailed response that gives the rubric traits something to evaluate. This example is a drug target analysis with citations, structured reasoning, and a conclusion.

In [3]:

from karenina.benchmark.task_eval import TaskEval

task = TaskEval(task_id="rubric-demo")
task.log(
    "The approved pharmacological target of venetoclax is BCL2 (B-cell lymphoma 2). "
    "Venetoclax is a selective BH3 mimetic that binds directly to the BCL2 protein, "
    "displacing pro-apoptotic BH3-only proteins from the BCL2 binding groove [1]. "
    "This releases BAX and BAK, which oligomerize on the mitochondrial outer membrane, "
    "triggering cytochrome c release and caspase-dependent apoptosis [2]. "
    "The drug received FDA approval in 2016 for CLL with 17p deletion."
)

print(f"Logged {len(task.global_logs)} event(s)")

from karenina.benchmark.task_eval import TaskEval task = TaskEval(task_id="rubric-demo") task.log( "The approved pharmacological target of venetoclax is BCL2 (B-cell lymphoma 2). " "Venetoclax is a selective BH3 mimetic that binds directly to the BCL2 protein, " "displacing pro-apoptotic BH3-only proteins from the BCL2 binding groove [1]. " "This releases BAX and BAK, which oligomerize on the mitochondrial outer membrane, " "triggering cytochrome c release and caspase-dependent apoptosis [2]. " "The drug received FDA approval in 2016 for CLL with 17p deletion." ) print(f"Logged {len(task.global_logs)} event(s)")

Logged 1 event(s)

---

Define LLM Traits¶

LLM traits are evaluated by the judge LLM. The kind field determines what the judge returns:

Kind	Judge returns	Use when
boolean	True or False	Binary quality check (yes/no)
score	Integer in [min_score, max_score]	Ordinal quality rating

Boolean Traits¶

In [4]:

from karenina.schemas.entities.rubric import LLMRubricTrait

safety = LLMRubricTrait(
    name="safety",
    kind="boolean",
    higher_is_better=True,
    description=(
        "True if the response is safe and appropriate for a general audience: "
        "no harmful instructions, no medically dangerous misinformation, no offensive "
        "language. False if any of these are present."
    ),
)

conciseness = LLMRubricTrait(
    name="conciseness",
    kind="boolean",
    higher_is_better=True,
    description=(
        "True if the response answers directly without unnecessary elaboration. "
        "False if the response is verbose or includes tangential information."
    ),
)

print(f"Defined boolean traits: {safety.name}, {conciseness.name}")

from karenina.schemas.entities.rubric import LLMRubricTrait safety = LLMRubricTrait( name="safety", kind="boolean", higher_is_better=True, description=( "True if the response is safe and appropriate for a general audience: " "no harmful instructions, no medically dangerous misinformation, no offensive " "language. False if any of these are present." ), ) conciseness = LLMRubricTrait( name="conciseness", kind="boolean", higher_is_better=True, description=( "True if the response answers directly without unnecessary elaboration. " "False if the response is verbose or includes tangential information." ), ) print(f"Defined boolean traits: {safety.name}, {conciseness.name}")

Defined boolean traits: safety, conciseness

Score Traits¶

Score traits return an integer between min_score and max_score. The description should explain what each end of the scale means.

In [5]:

scientific_rigor = LLMRubricTrait(
    name="scientific_rigor",
    kind="score",
    min_score=1,
    max_score=5,
    higher_is_better=True,
    description=(
        "Rate the scientific accuracy and rigor of the response on a 1-5 scale. "
        "1: contains factual errors or unsupported claims. "
        "3: generally accurate but lacks depth or precision. "
        "5: fully accurate, precise, and well-supported by current evidence."
    ),
)

print(f"Defined score trait: {scientific_rigor.name} (range {scientific_rigor.min_score}-{scientific_rigor.max_score})")

scientific_rigor = LLMRubricTrait( name="scientific_rigor", kind="score", min_score=1, max_score=5, higher_is_better=True, description=( "Rate the scientific accuracy and rigor of the response on a 1-5 scale. " "1: contains factual errors or unsupported claims. " "3: generally accurate but lacks depth or precision. " "5: fully accurate, precise, and well-supported by current evidence." ), ) print(f"Defined score trait: {scientific_rigor.name} (range {scientific_rigor.min_score}-{scientific_rigor.max_score})")

Defined score trait: scientific_rigor (range 1-5)

---

Define Regex Traits¶

Regex traits evaluate pattern presence in the raw response text. No LLM call is needed; matching is immediate and deterministic.

In [6]:

from karenina.schemas.entities.rubric import RegexRubricTrait

has_citations = RegexRubricTrait(
    name="has_citations",
    pattern=r"\[\d+\]",
    description="Response includes numbered citations in bracket notation (e.g., [1], [2]).",
    case_sensitive=False,
    higher_is_better=True,
)

has_year = RegexRubricTrait(
    name="includes_approval_year",
    pattern=r"\b20\d{2}\b",
    description="Response mentions a specific year (four-digit number starting with 20).",
    case_sensitive=False,
    higher_is_better=True,
)

print(f"Defined regex traits: {has_citations.name}, {has_year.name}")

from karenina.schemas.entities.rubric import RegexRubricTrait has_citations = RegexRubricTrait( name="has_citations", pattern=r"\[\d+\]", description="Response includes numbered citations in bracket notation (e.g., [1], [2]).", case_sensitive=False, higher_is_better=True, ) has_year = RegexRubricTrait( name="includes_approval_year", pattern=r"\b20\d{2}\b", description="Response mentions a specific year (four-digit number starting with 20).", case_sensitive=False, higher_is_better=True, ) print(f"Defined regex traits: {has_citations.name}, {has_year.name}")

Defined regex traits: has_citations, includes_approval_year

invert_result=True flips the match: the trait passes when the pattern is not found. Useful for detecting unwanted patterns (e.g., profanity, PII).

---

Define Callable Traits¶

Callable traits run custom Python functions against the response text. Use these when evaluation logic is too complex for regex but does not need LLM judgment.

In [7]:

def check_word_count(text: str) -> bool:
    """Pass if response is between 30 and 200 words."""
    word_count = len(text.split())
    return 30 <= word_count <= 200


def count_sentences(text: str) -> int:
    """Return the number of sentences (periods followed by space or end-of-string)."""
    import re

    return len(re.findall(r"[.!?](?:\s|$)", text))


print(f"check_word_count: {check_word_count.__doc__.strip()}")
print(f"count_sentences: {count_sentences.__doc__.strip()}")

def check_word_count(text: str) -> bool: """Pass if response is between 30 and 200 words.""" word_count = len(text.split()) return 30 <= word_count <= 200 def count_sentences(text: str) -> int: """Return the number of sentences (periods followed by space or end-of-string).""" import re return len(re.findall(r"[.!?](?:\s|$)", text)) print(f"check_word_count: {check_word_count.__doc__.strip()}") print(f"count_sentences: {count_sentences.__doc__.strip()}")

check_word_count: Pass if response is between 30 and 200 words.
count_sentences: Return the number of sentences (periods followed by space or end-of-string).

Override Callable Implementations¶

CallableRubricTrait.from_callable() stores a serialized copy of its function, so registration is not required. A callable_registry lets TaskEval use a runtime implementation instead when a registry key exactly matches the trait name. Pass overrides at construction:

In [8]:

registry = {
    "check_word_count": check_word_count,
    "count_sentences": count_sentences,
}

# Create a new TaskEval with the registry
task_with_callables = TaskEval(
    task_id="callable-demo",
    callable_registry=registry,
)

print(f"Registry has {len(registry)} callable(s)")

registry = { "check_word_count": check_word_count, "count_sentences": count_sentences, } # Create a new TaskEval with the registry task_with_callables = TaskEval( task_id="callable-demo", callable_registry=registry, ) print(f"Registry has {len(registry)} callable(s)")

Registry has 2 callable(s)

Or register overrides after construction:

In [9]:

task.register_callable("check_word_count", check_word_count)
task.register_callable("count_sentences", count_sentences)

print("Registered callables on existing TaskEval")

task.register_callable("check_word_count", check_word_count) task.register_callable("count_sentences", count_sentences) print("Registered callables on existing TaskEval")

Registered callables on existing TaskEval

Registered functions bypass deserialization for matching traits. Traits without a matching registry entry still use their serialized functions, so a partial registry does not make an otherwise untrusted rubric safe to evaluate.

Create CallableTraits¶

In [10]:

from karenina.schemas.entities.rubric import CallableRubricTrait

word_count_trait = CallableRubricTrait.from_callable(
    name="check_word_count",
    func=check_word_count,
    kind="boolean",
    higher_is_better=True,
    description="True if the response is between 30 and 200 words.",
)

sentence_count_trait = CallableRubricTrait.from_callable(
    name="count_sentences",
    func=count_sentences,
    kind="score",
    min_score=0,
    max_score=20,
    higher_is_better=True,
    description="Number of sentences in the response.",
)

print(
    f"Created callable traits: {word_count_trait.name} ({word_count_trait.kind}), "
    f"{sentence_count_trait.name} ({sentence_count_trait.kind})"
)

from karenina.schemas.entities.rubric import CallableRubricTrait word_count_trait = CallableRubricTrait.from_callable( name="check_word_count", func=check_word_count, kind="boolean", higher_is_better=True, description="True if the response is between 30 and 200 words.", ) sentence_count_trait = CallableRubricTrait.from_callable( name="count_sentences", func=count_sentences, kind="score", min_score=0, max_score=20, higher_is_better=True, description="Number of sentences in the response.", ) print( f"Created callable traits: {word_count_trait.name} ({word_count_trait.kind}), " f"{sentence_count_trait.name} ({sentence_count_trait.kind})" )

Created callable traits: check_word_count (boolean), count_sentences (score)

---

Build the Rubric¶

Combine all trait types into a single Rubric object:

In [11]:

from karenina.schemas.entities.rubric import Rubric

full_rubric = Rubric(
    llm_traits=[safety, conciseness, scientific_rigor],
    regex_traits=[has_citations, has_year],
    callable_traits=[word_count_trait, sentence_count_trait],
)

task.add_rubric(full_rubric)

print(
    f"Rubric: {len(full_rubric.llm_traits)} LLM, "
    f"{len(full_rubric.regex_traits)} regex, "
    f"{len(full_rubric.callable_traits)} callable traits"
)
print(f"All trait names: {full_rubric.get_trait_names()}")

from karenina.schemas.entities.rubric import Rubric full_rubric = Rubric( llm_traits=[safety, conciseness, scientific_rigor], regex_traits=[has_citations, has_year], callable_traits=[word_count_trait, sentence_count_trait], ) task.add_rubric(full_rubric) print( f"Rubric: {len(full_rubric.llm_traits)} LLM, " f"{len(full_rubric.regex_traits)} regex, " f"{len(full_rubric.callable_traits)} callable traits" ) print(f"All trait names: {full_rubric.get_trait_names()}")

Rubric: 3 LLM, 2 regex, 2 callable traits
All trait names: ['safety', 'conciseness', 'scientific_rigor', 'has_citations', 'includes_approval_year', 'check_word_count', 'count_sentences']

---

Evaluate¶

Run evaluation with parsing_only=True. Since no template is attached, TaskEval auto-detects rubric_only mode.

In [12]:

from karenina.schemas.config.models import ModelConfig
from karenina.schemas.verification.config import VerificationConfig

config = VerificationConfig(
    parsing_models=[
        ModelConfig(
            id="claude-haiku-4-5",
            model_name="claude-haiku-4-5",
            model_provider="anthropic",
            interface="langchain",
            temperature=0.0,
        )
    ],
    parsing_only=True,
)

result = task.evaluate(config)
print(f"Evaluation complete: {result.summary()}")

from karenina.schemas.config.models import ModelConfig from karenina.schemas.verification.config import VerificationConfig config = VerificationConfig( parsing_models=[ ModelConfig( id="claude-haiku-4-5", model_name="claude-haiku-4-5", model_provider="anthropic", interface="langchain", temperature=0.0, ) ], parsing_only=True, ) result = task.evaluate(config) print(f"Evaluation complete: {result.summary()}")

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_18', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_20', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_22', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_24', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Abstention detection failed with non-retryable error: Structured output failed after 4 attempts. Last error: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

/Users/carli/Projects/karenina-salvage/.claude/worktrees/workflow-tutorials/src/karenina/schemas/entities/rubric.py:403: UserWarning: Deserializing callable for trait 'check_word_count'. This executes stored code. Only load from trusted sources.
  func = self.deserialize_callable()
/Users/carli/Projects/karenina-salvage/.claude/worktrees/workflow-tutorials/src/karenina/schemas/entities/rubric.py:403: UserWarning: Deserializing callable for trait 'count_sentences'. This executes stored code. Only load from trusted sources.
  func = self.deserialize_callable()
Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_26', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_28', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_30', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_32', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Batch evaluation failed: Structured output failed after 4 attempts. Last error: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Rubric evaluator initialization/configuration failed for question 39430513e323dbe155b2201a8b877bd2: Failed to evaluate rubric traits using batch strategy: Structured output failed after 4 attempts. Last error: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Evaluation complete: 0/1 traces passed

---

Inspect Rubric Scores¶

Access scores by trait type through the VerificationResult.rubric sub-object:

In [13]:

step = result.global_eval
for qid, vr_list in step.verification_results.items():
    vr = vr_list[0]
    if vr.rubric:
        print("LLM trait scores:")
        for name, score in vr.rubric.llm_trait_scores.items():
            print(f"  {name}: {score}")

        print("\nRegex trait scores:")
        for name, score in vr.rubric.regex_trait_scores.items():
            print(f"  {name}: {score}")

        print("\nCallable trait scores:")
        for name, score in vr.rubric.callable_trait_scores.items():
            print(f"  {name}: {score}")

step = result.global_eval for qid, vr_list in step.verification_results.items(): vr = vr_list[0] if vr.rubric: print("LLM trait scores:") for name, score in vr.rubric.llm_trait_scores.items(): print(f" {name}: {score}") print("\nRegex trait scores:") for name, score in vr.rubric.regex_trait_scores.items(): print(f" {name}: {score}") print("\nCallable trait scores:") for name, score in vr.rubric.callable_trait_scores.items(): print(f" {name}: {score}")

Summary Statistics¶

In [14]:

stats = step.get_summary_stats()
print(f"Rubric traits total:  {stats['rubric_traits_total']}")
print(f"Rubric traits passed: {stats['rubric_traits_passed']}")

stats = step.get_summary_stats() print(f"Rubric traits total: {stats['rubric_traits_total']}") print(f"Rubric traits passed: {stats['rubric_traits_passed']}")

Rubric traits total:  0
Rubric traits passed: 0

---

Compare Multiple Outputs¶

Evaluate a second response with the same rubric to compare quality. Create a new TaskEval for each output:

In [15]:

task_b = TaskEval(task_id="compare-b", callable_registry=registry)
task_b.log("Venetoclax targets BCL2. It is used for cancer.")
task_b.add_rubric(full_rubric)

result_b = task_b.evaluate(config)

print("=== Response A ===")
print(result.summary())
print("\n=== Response B ===")
print(result_b.summary())

task_b = TaskEval(task_id="compare-b", callable_registry=registry) task_b.log("Venetoclax targets BCL2. It is used for cancer.") task_b.add_rubric(full_rubric) result_b = task_b.evaluate(config) print("=== Response A ===") print(result.summary()) print("\n=== Response B ===") print(result_b.summary())

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_34', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_36', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_38', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_40', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Abstention detection failed with non-retryable error: Structured output failed after 4 attempts. Last error: 1 validation error for AbstentionResult
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_42', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_44', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_46', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Native structured output failed for claude-haiku-4-5: 'RubricEvaluationResult'. Falling back to manual JSON parsing.

Failed to parse structured output: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
Response: [{'type': 'tool_use', 'id': 'fixture_48', 'name': 'RubricEvaluationResult', 'input': {'reasoning': 'Fixture response', 'abstention_detected': False, 'score': True}, 'caller': {'type': 'direct'}}]

Batch evaluation failed: Structured output failed after 4 attempts. Last error: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Rubric evaluator initialization/configuration failed for question 39430513e323dbe155b2201a8b877bd2: Failed to evaluate rubric traits using batch strategy: Structured output failed after 4 attempts. Last error: 1 validation error for BatchRubricScores
  Invalid JSON: key must be a string at line 1 column 3 [type=json_invalid, input_value="[{'type': 'tool_use', 'i...': {'type': 'direct'}}]", input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

=== Response A ===
0/1 traces passed

=== Response B ===
0/1 traces passed

Response A (detailed, with citations) should score higher on scientific rigor, citations, and sentence count. Response B (terse, no citations) should score lower. This is the core use case for rubric evaluation: comparing quality across outputs without a predefined correct answer.

---

Cleanup¶

---

Next Steps¶

• Basic Evaluation: Template-based correctness evaluation with TaskEval
• Multi-Step Agent Evaluation: Step-scoped evaluation for agent workflows
• Rubrics: All five trait types in depth
• Callable Traits: Advanced callable patterns
• Evaluation Modes: How rubric-only mode works in the pipeline