Skip to content
Karenina Documentation
Home
Initializing search
biocypher/karenina
Home
Getting Started
Core Concepts
Workflows
Reference
Advanced
Contributing
Karenina Documentation
biocypher/karenina
Home
Home
Philosophy
Reading Paths
Getting Started
Getting Started
Installation
Quick Start
Quick Start: Scenarios
Quick Start: TaskEval
Workspace Init
Agent Skills
Core Concepts
Core Concepts
Questions & Benchmarks
Questions & Benchmarks
Benchmarks: The Reproducible Package
Questions: The Heart of Evaluation
Checkpoints: The Memory of Evaluation
TaskEval
Answer Templates
Verification Primitives
Rubric Traits
Rubric Traits
LLM Rubric Traits
Regex Traits
Callable Traits
Metric Traits
Agentic Rubric Traits
Scenarios
Scenarios
Building Scenarios
Outcome Criteria
State and Routing
Handover (Multi-Agent Routing)
Execution
Templates vs Rubrics
Evaluation Modes
Results and Scoring
Question Context
Question Context
ADeLe: Question Classification
Few-Shot Examples
Pipeline
Pipeline
Verification Pipeline
Prompt Assembly
Agentic Evaluation
Extending Prior Runs: and
Infrastructure
Infrastructure
Adapters
MCP Integration
Manual Interface
Workflows
Workflows
Evaluating with TaskEval
Evaluating with TaskEval
Basic TaskEval Evaluation
Multi-Step Agent Evaluation
Quality Assessment with Rubrics
Configuration
Configuration
Environment Variables
Presets
Creating Benchmarks
Creating Benchmarks
Factual QA Benchmark
Full Evaluation Benchmark
Quality Assessment Benchmark
Choosing the Right Rubric Trait Type
Scaled Benchmark Creation
AI-Assisted Template Construction
ADeLe Question Classification
Benchmark Operations
Running Verification
Running Verification
Basic Verification
Full Evaluation
Multi-Model Comparison
Deep Judgment
MCP Agent Evaluation
Manual Interface
Agentic Evaluation
Extending a Prior Run
Few-Shot Example Configuration
Progressive Save and Resume
Analyzing Results
Analyzing Results
Verification Result
DataFrame Analysis
Database Persistence
Error Analysis
Exporting
Iterating
Scenarios
Scenarios
Sycophancy Detection: A Scenario Tutorial
Library Walkthrough
Library Walkthrough
Defining a Benchmark
Adding Questions to a Benchmark
Accessing and Filtering Questions
Answer Templates
Rubrics
Model Configuration
Running Verification
Analyzing Verification Results with DataFrames
Saving and Loading Benchmarks
Reference
Reference
Configuration Reference
Configuration Reference
Environment Variables
VerificationConfig
ModelConfig
PromptConfig
Preset Schema
CLI Reference
CLI Reference
verify
verify-status
preset
serve
init
optimize
Failure and Caveats
Sinks
Replay
API Reference
API Reference
Core API
Core API
karenina
karenina
karenina
adapters
adapters
karenina.adapters
agent_runtime
claude_agent_sdk
claude_tool
factory
langchain
langchain_deep_agents
llm_parallel
manual
registry
taskeval
benchmark
benchmark
karenina.benchmark
authoring
benchmark
benchmark_helpers
core
curation_workbook
error_analysis
run_artifacts
task_eval
trace
verification
cli
exceptions
integrations
ports
ports
karenina.ports
adapter_instruction
agent
capabilities
errors
llm
messages
parser
usage
replay
scenario
schemas
schemas
karenina.schemas
checkpoint
config
config
karenina.schemas.config
models
dataframes
dataframes
karenina.schemas.dataframes
judgment
rubric
scenario
template
entities
entities
karenina.schemas.entities
answer
composition
conditional
question
rubric
template_spec
verified_field
outputs
outputs
karenina.schemas.outputs
rubric
primitives
results
results
karenina.schemas.results
aggregation
caveat
failure
judgment
row_key
rubric
rubric_judgment
scenario
template
verification_result_set
scenario
shared
verification
verification
karenina.schemas.verification
api_models
config
config_presets
job
model_identity
prompt_config
result
result_components
storage
utils
Internals
Internals
karenina
karenina
adapters
adapters
claude_agent_sdk
claude_agent_sdk
agent
auth
availability
docker_cli_wrapper
errors
llm
mcp
messages
parser
prompts
prompts
karenina.adapters.claude_agent_sdk.prompts
abstention
deep_judgment
parsing
rubric
sufficiency
registration
trace
usage
claude_tool
claude_tool
agent
llm
mcp
messages
parser
prompts
prompts
karenina.adapters.claude_tool.prompts
abstention
deep_judgment
parsing
rubric
sufficiency
registration
tools
trace
usage
langchain
langchain
agent
initialization
llm
mcp
messages
middleware
models
parser
prompts
prompts
karenina.adapters.langchain.prompts
abstention
deep_judgment
feedback_format
feedback_null
format_instructions
parsing
rubric
sufficiency
summarization
registration
trace
usage
langchain_deep_agents
langchain_deep_agents
agent
availability
docker_backend
errors
initialization
llm
mcp
messages
parser
prompts
prompts
karenina.adapters.langchain_deep_agents.prompts
abstention
deep_judgment
parsing
rubric
sufficiency
read_only_backend
registration
runtime
trace
usage
manual
manual
exceptions
helpers
manager
message_utils
registration
traces
taskeval
taskeval
registration
benchmark
benchmark
authoring
authoring
answers
answers
karenina.benchmark.authoring.answers
builder
generator
generator_code
generator_prompts
reader
questions
questions
karenina.benchmark.authoring.questions
extractor
reader
core
core
base
exports
metadata
question_query
questions
results
results_io
results_store
results_stream
rubrics
templates
verification_manager
error_analysis
error_analysis
case_renderer
exceptions
facade
indexer
launcher
launchers
launchers
karenina.benchmark.error_analysis.launchers
claude_code
prepare_only
materializer
prompt_io
prompts
task_eval
task_eval
helpers
models
task_eval
verification
verification
async_lifecycle
batch_runner
evaluators
evaluators
karenina.benchmark.verification.evaluators
rubric
rubric
karenina.benchmark.verification.evaluators.rubric
agentic_trait
deep_judgment
evaluator
llm_trait
metric_trait
template
template
karenina.benchmark.verification.evaluators.template
deep_judgment
evaluator
results
trace
trace
karenina.benchmark.verification.evaluators.trace
abstention
sufficiency
executor
extension
failure_classifier
model_stamping
portal_pool
post_hoc
prompts
prompts
karenina.benchmark.verification.prompts
assembler
deep_judgment
deep_judgment
karenina.benchmark.verification.prompts.deep_judgment
rubric
rubric
karenina.benchmark.verification.prompts.deep_judgment.rubric
deep_judgment
template
template
karenina.benchmark.verification.prompts.deep_judgment.template
excerpt
hallucination
reasoning
reasoning_only
parsing
parsing
karenina.benchmark.verification.prompts.parsing
dynamic_decision
parsing_instructions
rubric
rubric
karenina.benchmark.verification.prompts.rubric
literal_trait
llm_trait
metric_trait
presence_check
task_types
trace
trace
karenina.benchmark.verification.prompts.trace
abstention
sufficiency
repair
runner
scenario_executor
sinks
stages
stages
karenina.benchmark.verification.stages
core
core
karenina.benchmark.verification.stages.core
autofail_stage_base
base
check_stage_base
orchestrator
helpers
helpers
karenina.benchmark.verification.stages.helpers
agentic_parse_helpers
deep_judgment_helpers
results_exporter
pipeline
pipeline
karenina.benchmark.verification.stages.pipeline
abstention_check
agentic_parse_template
agentic_rubric_evaluation
deep_judgment_autofail
deep_judgment_rubric_auto_fail
dynamic_parse_template
embedding_check
finalize_result
generate_answer
parse_template
placeholder_retry_autofail
recursion_limit_autofail
rubric_evaluation
sufficiency_check
trace_validation_autofail
validate_template
verify_template
utils
utils
karenina.benchmark.verification.utils
cache_helpers
class_discovery
embedding_check
error_helpers
llm_invocation
llm_judge_helpers
parser_resilience
resource_helpers
schema_builder
search_helpers
search_provider
search_tavily
storage_helpers
task_helpers
template_parsing_helpers
template_validation
token_budget
trace_agent_metrics
trace_formatting
trace_fuzzy_match
trace_masking
trace_parsing
trace_usage_tracker
usage_helpers
workspace_capture
cli
cli
analyze_errors
interactive
optimize
preset
repair
serve
status
utils
verify
verify_config
verify_output
integrations
integrations
adele
adele
karenina.integrations.adele
classifier
parser
prompts
prompts
karenina.integrations.adele.prompts
system
user
schemas
traits
gepa
gepa
karenina.integrations.gepa
adapter
config
data_types
export
feedback
prompts
prompts
karenina.integrations.gepa.prompts
defaults
feedback
scoring
splitting
tracking
verbose
replay
replay
capture
exceptions
persistence
ports_message_hydration
projection
store
scenario
scenario
builder
checkpoint
edge_resolution
handover
manager
outcome_evaluation
source_extraction
sugar
trace_materialization
validation
schemas
schemas
primitives
primitives
comparisons
composition
normalizers
registry
scope
trace
scenario
scenario
checks
definition
state
types
shared
shared
search
storage
storage
auto_mapper
base
converters
db_config
engine
generated_models
migrate_question_keywords
migrate_template_id
models
operations
queries
rubric_serializer
utils
views
views
karenina.storage.views
combination_info
deep_judgment_rubric_traits
models_used
question_attributes
raw_llm_answers
result_mcp_servers
result_tools_used
results_metadata
rubric_traits
template_attributes
template_results
utils
utils
utils
answer_cache
checkpoint
checkpoint_trait_converters
code
endpoint_probe
errors
file_ops
json_extraction
mcp
mcp
karenina.utils.mcp
client
server
tools
messages
progressive_save
retry_policy
testing
version
Advanced
Advanced
Advanced
Pipeline
Pipeline
Stages
Deep Judgment (Templates)
Deep Judgment (Rubrics)
Prompt Assembly
Custom Stages
Agentic Evaluation
Agentic Rubric Evaluation
Error Handling and Retries
Replay Store
Stored Results, Run Directories, and Repair
Adapters
Adapters
Ports
Available Adapters
MCP Integration
Writing Adapters
Features
Features
Abstention Detection
Deep-Judgment Parsing
Embedding Check (Semantic Fallback)
Few-Shot Prompting
Manual Trace System
MCP Integration
Configuration Presets
TaskEval: Task-Centric Trace Evaluation
GEPA Optimization
GEPA Optimization
GEPA Integration Quick Start
GEPA Configuration Deep Dive
Data Splitting for GEPA Optimization
Scoring Deep Dive
KareninaAdapter: GEPA Integration
LLM Feedback Generation
Optimization Run Tracking
Export and Reuse Optimized Prompts
Full GEPA Optimization Workflow
MCP Tool Description Optimization
Verbose Real GEPA Optimization on AIME
Contributing
Contributing
Contributing
Karenina Documentation
¶
Redirecting to
What is Karenina?
...
Back to top