Quick Start: TaskEval¶

TaskEval evaluates textual outputs that come from an external source: an LLM response, a paper excerpt, a chat transcript, or any other text you already have. You supply the text; Karenina's judge LLM handles the evaluation. This guide walks you through logging outputs, attaching evaluation criteria, running evaluation, and inspecting results.

By the end you will have evaluated a pre-recorded LLM response for both correctness (via an answer template) and quality (via rubric traits), with no question definition or answer generation required.

---

Prerequisites¶

• Python 3.11+
• Karenina installed (see Installation)
• API key for the judge LLM provider:

export ANTHROPIC_API_KEY="sk-ant-..."

---

Step 1: Create a TaskEval Instance¶

TaskEval is a recording and evaluation container. You log text into it, attach evaluation criteria, then run evaluation. Create one with an optional task ID and metadata for tracking.

In [2]:

from karenina.benchmark.task_eval import TaskEval

task = TaskEval(
    task_id="drug-target-eval",
    metadata={"model": "claude-haiku-4-5", "scenario": "pharmacology"},
)

print(f"Created TaskEval: {task.task_id}")

from karenina.benchmark.task_eval import TaskEval task = TaskEval( task_id="drug-target-eval", metadata={"model": "claude-haiku-4-5", "scenario": "pharmacology"}, ) print(f"Created TaskEval: {task.task_id}")

Created TaskEval: drug-target-eval

Learn more: TaskEval Concepts

---

Step 2: Log the Output to Evaluate¶

TaskEval evaluates text that you supply. Use log() to record any string as content for evaluation.

Here we log an LLM response about a drug target. This could come from any source: an agent run, a CI pipeline, an external API, or a manual experiment.

In [3]:

task.log(
    "The approved drug target of venetoclax is BCL2 (B-cell lymphoma 2). "
    "Venetoclax is a selective BCL2 inhibitor that works by displacing pro-apoptotic "
    "proteins, triggering programmed cell death in cancer cells [1]."
)

print(f"Logged {len(task.global_logs)} event(s)")

task.log( "The approved drug target of venetoclax is BCL2 (B-cell lymphoma 2). " "Venetoclax is a selective BCL2 inhibitor that works by displacing pro-apoptotic " "proteins, triggering programmed cell death in cancer cells [1]." ) print(f"Logged {len(task.global_logs)} event(s)")

Logged 1 event(s)

Learn more: Logging methods · Structured trace logging

---

Step 3: Define an Answer Template¶

An answer template is a Pydantic schema that defines what to extract from the logged output and how to verify it. Each field uses VerifiedField to declare what to extract, the correct value, and a verification primitive that checks the result.

In [4]:

from karenina.schemas.entities import BaseAnswer, VerifiedField
from karenina.schemas.primitives import BooleanMatch


class Answer(BaseAnswer):
    identifies_bcl2_as_target: bool = VerifiedField(
        description=(
            "True if the response identifies BCL2 (including Bcl-2, BCL-2, or "
            "B-cell lymphoma 2) as the direct pharmacological target of venetoclax. "
            "False if BCL2 is mentioned only as a pathway member or a different "
            "protein is identified as the primary target."
        ),
        ground_truth=True,
        verify_with=BooleanMatch(),
    )
    mentions_mechanism: bool = VerifiedField(
        description=(
            "True if the response explains the mechanism of action (e.g., inhibiting "
            "BCL2 to trigger apoptosis). False if only the target is named without "
            "any mechanistic explanation."
        ),
        ground_truth=True,
        verify_with=BooleanMatch(),
    )


task.add_template(Answer)

print("Attached answer template with 2 verification fields")

from karenina.schemas.entities import BaseAnswer, VerifiedField from karenina.schemas.primitives import BooleanMatch class Answer(BaseAnswer): identifies_bcl2_as_target: bool = VerifiedField( description=( "True if the response identifies BCL2 (including Bcl-2, BCL-2, or " "B-cell lymphoma 2) as the direct pharmacological target of venetoclax. " "False if BCL2 is mentioned only as a pathway member or a different " "protein is identified as the primary target." ), ground_truth=True, verify_with=BooleanMatch(), ) mentions_mechanism: bool = VerifiedField( description=( "True if the response explains the mechanism of action (e.g., inhibiting " "BCL2 to trigger apoptosis). False if only the target is named without " "any mechanistic explanation." ), ground_truth=True, verify_with=BooleanMatch(), ) task.add_template(Answer) print("Attached answer template with 2 verification fields")

Attached answer template with 2 verification fields

Learn more: Answer Templates · Template Construction

---

Step 4: Add Rubric Traits¶

While templates verify correctness, rubrics assess quality. Each trait evaluates one dimension of the response independently. Here we add an LLM-judged score trait and a regex pattern trait.

In [5]:

from karenina.schemas.entities.rubric import LLMRubricTrait, RegexRubricTrait, Rubric

rubric = Rubric(
    llm_traits=[
        LLMRubricTrait(
            name="Conciseness",
            description=(
                "Rate how concise the response is on a scale of 1-5, where "
                "1 is very verbose and 5 is extremely concise."
            ),
            kind="score",
        ),
    ],
    regex_traits=[
        RegexRubricTrait(
            name="Has Citations",
            description="The response includes numbered citations in bracket notation (e.g., [1]).",
            pattern=r"\[\d+\]",
            case_sensitive=False,
        ),
    ],
)

task.add_rubric(rubric)

print(f"Added rubric with {len(rubric.llm_traits)} LLM trait(s) and {len(rubric.regex_traits)} regex trait(s)")

from karenina.schemas.entities.rubric import LLMRubricTrait, RegexRubricTrait, Rubric rubric = Rubric( llm_traits=[ LLMRubricTrait( name="Conciseness", description=( "Rate how concise the response is on a scale of 1-5, where " "1 is very verbose and 5 is extremely concise." ), kind="score", ), ], regex_traits=[ RegexRubricTrait( name="Has Citations", description="The response includes numbered citations in bracket notation (e.g., [1]).", pattern=r"\[\d+\]", case_sensitive=False, ), ], ) task.add_rubric(rubric) print(f"Added rubric with {len(rubric.llm_traits)} LLM trait(s) and {len(rubric.regex_traits)} regex trait(s)")

Added rubric with 1 LLM trait(s) and 1 regex trait(s)

Learn more: Rubrics · All trait types: LLM, Regex, Callable, Metric, Agentic

---

Step 5: Run Evaluation¶

Configure the judge LLM and run evaluation. TaskEval uses parsing_only=True since no answering model is needed.

In [6]:

from karenina.schemas.config.models import ModelConfig
from karenina.schemas.verification.config import VerificationConfig

config = VerificationConfig(
    parsing_models=[
        ModelConfig(
            id="claude-haiku-4-5",
            model_name="claude-haiku-4-5",
            model_provider="anthropic",
            interface="langchain",
            temperature=0.0,
        )
    ],
    parsing_only=True,
)

result = task.evaluate(config)

print(f"Evaluation complete: {result.summary()}")

from karenina.schemas.config.models import ModelConfig from karenina.schemas.verification.config import VerificationConfig config = VerificationConfig( parsing_models=[ ModelConfig( id="claude-haiku-4-5", model_name="claude-haiku-4-5", model_provider="anthropic", interface="langchain", temperature=0.0, ) ], parsing_only=True, ) result = task.evaluate(config) print(f"Evaluation complete: {result.summary()}")

Evaluation complete: 1/1 template verifications passed | 2/2 rubric traits passed

Learn more: Evaluation modes · Model Config

---

Step 6: Inspect Results¶

evaluate() returns a TaskEvalResult. The quickest way to see what happened is display(), which prints a formatted report with template pass/fail status and rubric scores:

In [7]:

print(result.display())

print(result.display())

════════════════════════════════════════════════════════════════════════════════
                        TASK EVALUATION RESULTS
════════════════════════════════════════════════════════════════════════════════
Task ID: drug-target-eval
Model: claude-haiku-4-5
Scenario: pharmacology
Timestamp: 2026-03-05 11:47:10

────────────────────────────────────────────────────────────
GLOBAL EVALUATION
────────────────────────────────────────────────────────────
Verification Results:
  Question: answer
     Status: ✓ PASSED
     Output: "--- AI Message ---
The approved drug target of venetoclax is BCL2 (B-cell lymphoma 2). Venetoclax is a selective BCL2 inhibitor that works by displacing pro-apoptotic proteins, triggering programmed cell death in cancer cells [1]."
     Rubric: Conciseness=4, Has Citations=✓

════════════════════════════════════════════════════════════════════════════════
SUMMARY: 1/1 template verifications passed | 2/2 rubric traits passed
════════════════════════════════════════════════════════════════════════════════

For a one-line overview, use summary():

In [8]:

print(result.summary())

print(result.summary())

1/1 template verifications passed | 2/2 rubric traits passed

You can also export the full results as JSON or Markdown for downstream analysis or archiving:

In [9]:

print(result.export_json()[:300])

print(result.export_json()[:300])

{
  "task_id": "drug-target-eval",
  "metadata": {
    "model": "claude-haiku-4-5",
    "scenario": "pharmacology"
  },
  "timestamp": "2026-03-05T11:47:10.909758",
  "summary": "1/1 template verifications passed | 2/2 rubric traits passed",
  "global_evaluation": {
    "verification_results": {
   

For programmatic access to individual fields (template verdicts, rubric scores, metadata), see the results inspection reference.

Learn more: Results inspection · DataFrame analysis

---

Bonus: Multi-Step Evaluation¶

TaskEval supports step-scoped evaluation for multi-phase agent workflows. Each step gets its own logs, templates, and rubric traits, evaluated independently.

In [10]:

multi_task = TaskEval(task_id="multi-step-agent")

# Log per-step outputs
multi_task.log(
    "Found 3 relevant papers on venetoclax mechanism of action.",
    step_id="retrieval",
)
multi_task.log(
    "BCL2 is the primary target of venetoclax. It selectively inhibits BCL2, triggering apoptosis in CLL cells.",
    step_id="synthesis",
)

# Step-specific rubrics
multi_task.add_rubric(
    Rubric(
        llm_traits=[
            LLMRubricTrait(
                name="retrieval_quality",
                kind="boolean",
                description="True if relevant sources were found for the query.",
            )
        ]
    ),
    step_id="retrieval",
)
multi_task.add_rubric(
    Rubric(
        llm_traits=[
            LLMRubricTrait(
                name="synthesis_accuracy",
                kind="boolean",
                description="True if the synthesis accurately reflects the retrieved information.",
            )
        ]
    ),
    step_id="synthesis",
)

# Evaluate each step individually
for step_id in ["retrieval", "synthesis"]:
    step_result = multi_task.evaluate(config, step_id=step_id)
    step_eval = step_result.per_step[step_id]
    stats = step_eval.get_summary_stats()
    print(f"Step '{step_id}': {stats['rubric_traits_passed']}/{stats['rubric_traits_total']} traits passed")

multi_task = TaskEval(task_id="multi-step-agent") # Log per-step outputs multi_task.log( "Found 3 relevant papers on venetoclax mechanism of action.", step_id="retrieval", ) multi_task.log( "BCL2 is the primary target of venetoclax. It selectively inhibits BCL2, triggering apoptosis in CLL cells.", step_id="synthesis", ) # Step-specific rubrics multi_task.add_rubric( Rubric( llm_traits=[ LLMRubricTrait( name="retrieval_quality", kind="boolean", description="True if relevant sources were found for the query.", ) ] ), step_id="retrieval", ) multi_task.add_rubric( Rubric( llm_traits=[ LLMRubricTrait( name="synthesis_accuracy", kind="boolean", description="True if the synthesis accurately reflects the retrieved information.", ) ] ), step_id="synthesis", ) # Evaluate each step individually for step_id in ["retrieval", "synthesis"]: step_result = multi_task.evaluate(config, step_id=step_id) step_eval = step_result.per_step[step_id] stats = step_eval.get_summary_stats() print(f"Step '{step_id}': {stats['rubric_traits_passed']}/{stats['rubric_traits_total']} traits passed")

AbstentionCheck triggered for question c2a195f75adcf3e4c69280d908b1c7b4 - overriding result to False

Step 'retrieval': 1/1 traits passed
Step 'synthesis': 1/1 traits passed

Learn more: Multi-step evaluation · Step scoping

---

Next Steps¶

• TaskEval Concepts: Merge strategies, object structure, pipeline integration
• Answer Templates: Field types, descriptions, and writing verify() methods
• Rubrics: All five trait types (LLM, Regex, Callable, Metric, Agentic)
• Benchmark Quick Start: If you need Karenina to generate responses and evaluate them