Deep-Judgment Parsing¶

Deep-judgment is an advanced parsing mode that provides enhanced transparency and accountability by extracting verbatim evidence from LLM responses before drawing conclusions. This guide explains what it is, when to use it, and how to configure it.

What is Deep-Judgment?¶

Deep-judgment parsing is a multi-stage evaluation process that goes beyond standard template-based verification. Instead of directly extracting structured data from LLM responses, deep-judgment performs a three-stage analysis:

1. Excerpt Extraction: Identifies verbatim quotes that support each answer attribute
2. Reasoning Generation: Explains how the excerpts map to attribute values
3. Parameter Extraction: Extracts final structured values with full context

This approach creates an audit trail showing exactly what evidence the LLM provided and how it was interpreted.

Standard Parsing vs Deep-Judgment¶

Standard Parsing (default):

LLM Response → Parse Attributes → Verify Correctness

Deep-Judgment Parsing:

LLM Response → Extract Excerpts → Generate Reasoning → Parse Attributes → Verify Correctness

Why Use Deep-Judgment?¶

1. Transparency¶

Every extracted attribute is backed by explicit evidence from the LLM response. You can see exactly which parts of the answer support each claim.

Example:

Question: "What is the approved drug target of Venetoclax?"

LLM Response: "Venetoclax targets the BCL-2 protein, which is an anti-apoptotic
protein. By inhibiting BCL-2, venetoclax promotes apoptosis in cancer cells."

Standard Parsing:
  drug_target: "BCL-2" ✓

Deep-Judgment Parsing:
  Excerpt: "targets the BCL-2 protein"
  Reasoning: "The response explicitly states BCL-2 as the target protein"
  drug_target: "BCL-2" ✓

How Deep-Judgment Works¶

Deep-judgment uses a three-stage autoregressive process where each stage builds on the previous one:

Stage 1: Excerpt Extraction¶

The parsing model identifies verbatim quotes from the LLM response that support each template attribute.

For each attribute:

• Extract 0-3 excerpts (configurable)
• Assign confidence level: low/medium/high
• Validate excerpts actually exist in the response (fuzzy matching)
• If no excerpts found, request explanation from LLM

Example:

# Question: "What is the approved drug target of Venetoclax?"
# Response: "Venetoclax targets BCL-2, a key anti-apoptotic protein"

excerpts = {
    "drug_target": [
        {
            "text": "targets BCL-2",
            "confidence": "high",
            "similarity_score": 0.95
        }
    ]
}

In [ ]:

# Example: Extracted excerpts structure
excerpts = {"drug_target": [{"text": "targets BCL-2", "confidence": "high", "similarity_score": 0.95}]}
print("Extracted Excerpts:")
for attr, exc_list in excerpts.items():
    print(f"\nAttribute: {attr}")
    for exc in exc_list:
        print(f"  Text: {exc['text']}")
        print(f"  Confidence: {exc['confidence']}")
        print(f"  Similarity: {exc['similarity_score']}")

# Example: Extracted excerpts structure excerpts = {"drug_target": [{"text": "targets BCL-2", "confidence": "high", "similarity_score": 0.95}]} print("Extracted Excerpts:") for attr, exc_list in excerpts.items(): print(f"\nAttribute: {attr}") for exc in exc_list: print(f" Text: {exc['text']}") print(f" Confidence: {exc['confidence']}") print(f" Similarity: {exc['similarity_score']}")

Stage 2: Reasoning Generation¶

The parsing model explains how the excerpts from Stage 1 inform each attribute value.

Example:

reasoning = {
    "drug_target": "The excerpt 'targets BCL-2' explicitly states BCL-2
                  as the protein target. This directly answers the question."
}

In [ ]:

# Example: Attribute reasoning
reasoning = {
    "drug_target": "The excerpt 'targets BCL-2' explicitly states BCL-2 as the protein target.",
    "mechanism": "The response explains that venetoclax inhibits BCL-2, which promotes apoptosis in cancer cells.",
}
print("Attribute Reasoning:")
for attr, reason in reasoning.items():
    print(f"\n{attr}: {reason}")

# Example: Attribute reasoning reasoning = { "drug_target": "The excerpt 'targets BCL-2' explicitly states BCL-2 as the protein target.", "mechanism": "The response explains that venetoclax inhibits BCL-2, which promotes apoptosis in cancer cells.", } print("Attribute Reasoning:") for attr, reason in reasoning.items(): print(f"\n{attr}: {reason}")

Stage 3: Parameter Extraction¶

Using the reasoning context from Stage 2, the parsing model extracts structured attribute values using standard template parsing.

Example:

parsed_answer = {
    "drug_target": "BCL-2"
}

Validation and Auto-Fail¶

If any attribute has missing excerpts (no verbatim evidence found), verification automatically fails even if the final parsed answer seems correct. This ensures all claims are backed by explicit evidence.

Exception: If abstention is detected (LLM refused to answer), auto-fail is skipped since abstention takes priority.

Enabling Deep-Judgment¶

Deep-judgment is disabled by default. Enable it in your verification configuration:

In [ ]:

from karenina import Benchmark
from karenina.schemas import ModelConfig, VerificationConfig

# Configure models
model_config = ModelConfig(
    id="gpt-4.1-mini", model_provider="openai", model_name="gpt-4.1-mini", temperature=0.0, interface="langchain"
)

# Enable deep-judgment
config = VerificationConfig(
    answering_models=[model_config],
    parsing_models=[model_config],
    deep_judgment_enabled=True,  # Enable deep-judgment parsing
)

print("Deep-judgment enabled:", config.deep_judgment_enabled)

from karenina import Benchmark from karenina.schemas import ModelConfig, VerificationConfig # Configure models model_config = ModelConfig( id="gpt-4.1-mini", model_provider="openai", model_name="gpt-4.1-mini", temperature=0.0, interface="langchain" ) # Enable deep-judgment config = VerificationConfig( answering_models=[model_config], parsing_models=[model_config], deep_judgment_enabled=True, # Enable deep-judgment parsing ) print("Deep-judgment enabled:", config.deep_judgment_enabled)

Advanced Configuration¶

You can tune deep-judgment behavior with additional parameters:

In [ ]:

config = VerificationConfig(
    answering_models=[model_config],
    parsing_models=[model_config],
    # Deep-judgment configuration
    deep_judgment_enabled=True,  # Enable feature
    deep_judgment_max_excerpts_per_attribute=3,  # Excerpts per attribute (1-5)
    deep_judgment_fuzzy_match_threshold=0.80,  # Similarity threshold (0.0-1.0)
    deep_judgment_excerpt_retry_attempts=2,  # Retry attempts (0-5)
)

print("Deep-Judgment Configuration:")
print(f"  Enabled: {config.deep_judgment_enabled}")
print(f"  Max excerpts per attribute: {config.deep_judgment_max_excerpts_per_attribute}")
print(f"  Fuzzy match threshold: {config.deep_judgment_fuzzy_match_threshold}")
print(f"  Retry attempts: {config.deep_judgment_excerpt_retry_attempts}")

config = VerificationConfig( answering_models=[model_config], parsing_models=[model_config], # Deep-judgment configuration deep_judgment_enabled=True, # Enable feature deep_judgment_max_excerpts_per_attribute=3, # Excerpts per attribute (1-5) deep_judgment_fuzzy_match_threshold=0.80, # Similarity threshold (0.0-1.0) deep_judgment_excerpt_retry_attempts=2, # Retry attempts (0-5) ) print("Deep-Judgment Configuration:") print(f" Enabled: {config.deep_judgment_enabled}") print(f" Max excerpts per attribute: {config.deep_judgment_max_excerpts_per_attribute}") print(f" Fuzzy match threshold: {config.deep_judgment_fuzzy_match_threshold}") print(f" Retry attempts: {config.deep_judgment_excerpt_retry_attempts}")

Configuration Parameters:

Parameter	Type	Default	Description
deep_judgment_enabled	bool	False	Enable/disable deep-judgment parsing
deep_judgment_max_excerpts_per_attribute	int	3	Maximum excerpts to extract per attribute (1-5)
deep_judgment_fuzzy_match_threshold	float	0.80	Similarity threshold for excerpt validation (0.0-1.0). Higher = stricter.
deep_judgment_excerpt_retry_attempts	int	2	Retry attempts when excerpt validation fails (0-5)

Search-Enhanced Deep-Judgment¶

Search-enhanced deep-judgment extends the standard three-stage process with an additional validation layer that checks extracted excerpts against external evidence sources. This helps detect potential hallucinations by verifying that the information in excerpts can be corroborated by external search results.

In [ ]:

# Enable search-enhanced deep-judgment
config = VerificationConfig(
    answering_models=[model_config],
    parsing_models=[model_config],
    deep_judgment_enabled=True,  # Enable deep-judgment
    deep_judgment_search_enabled=True,  # Enable search validation
    deep_judgment_search_tool="tavily",  # Use Tavily search (default)
)

print("Search-Enhanced Deep-Judgment:")
print(f"  Deep-judgment enabled: {config.deep_judgment_enabled}")
print(f"  Search enabled: {config.deep_judgment_search_enabled}")
print(f"  Search tool: {config.deep_judgment_search_tool}")

# Enable search-enhanced deep-judgment config = VerificationConfig( answering_models=[model_config], parsing_models=[model_config], deep_judgment_enabled=True, # Enable deep-judgment deep_judgment_search_enabled=True, # Enable search validation deep_judgment_search_tool="tavily", # Use Tavily search (default) ) print("Search-Enhanced Deep-Judgment:") print(f" Deep-judgment enabled: {config.deep_judgment_enabled}") print(f" Search enabled: {config.deep_judgment_search_enabled}") print(f" Search tool: {config.deep_judgment_search_tool}")

Understanding Results¶

When deep-judgment is enabled, verification results include additional metadata about the extraction process.

In [ ]:

# Create a mock result to show the structure
mock_result = create_mock_verification_result(
    question_id="q1", question_text="What is the approved drug target of Venetoclax?", deep_judgment=True
)

# Access deep-judgment results
if mock_result.deep_judgment_performed:
    print("Deep-Judgment Results:")
    print("\nMetadata:")
    print(f"  Question: {mock_result.question_text}")
    print(f"  Stages Completed: {mock_result.deep_judgment_stages_completed}")
    print(f"  Model Calls: {mock_result.deep_judgment_model_calls}")
    print(f"  Excerpt Retries: {mock_result.deep_judgment_excerpt_retry_count}")

# Create a mock result to show the structure mock_result = create_mock_verification_result( question_id="q1", question_text="What is the approved drug target of Venetoclax?", deep_judgment=True ) # Access deep-judgment results if mock_result.deep_judgment_performed: print("Deep-Judgment Results:") print("\nMetadata:") print(f" Question: {mock_result.question_text}") print(f" Stages Completed: {mock_result.deep_judgment_stages_completed}") print(f" Model Calls: {mock_result.deep_judgment_model_calls}") print(f" Excerpt Retries: {mock_result.deep_judgment_excerpt_retry_count}")

In [ ]:

# Display extracted excerpts
if mock_result.extracted_excerpts:
    print("\nExtracted Excerpts:")
    for attr, excerpts in mock_result.extracted_excerpts.items():
        print(f"\n  Attribute: {attr}")
        for exc in excerpts:
            if exc.get("explanation"):
                # Missing excerpt with explanation
                print(f"    [Missing] {exc['explanation']}")
            else:
                # Found excerpt
                print(f'    Text: "{exc["text"]}"')
                print(f"    Confidence: {exc['confidence']}")
                print(f"    Similarity: {exc['similarity_score']:.2f}")

# Display extracted excerpts if mock_result.extracted_excerpts: print("\nExtracted Excerpts:") for attr, excerpts in mock_result.extracted_excerpts.items(): print(f"\n Attribute: {attr}") for exc in excerpts: if exc.get("explanation"): # Missing excerpt with explanation print(f" [Missing] {exc['explanation']}") else: # Found excerpt print(f' Text: "{exc["text"]}"') print(f" Confidence: {exc['confidence']}") print(f" Similarity: {exc['similarity_score']:.2f}")

In [ ]:

# Display reasoning traces
if mock_result.attribute_reasoning:
    print("\nReasoning:")
    for attr, reasoning in mock_result.attribute_reasoning.items():
        print(f"  {attr}: {reasoning}")

# Display reasoning traces if mock_result.attribute_reasoning: print("\nReasoning:") for attr, reasoning in mock_result.attribute_reasoning.items(): print(f" {attr}: {reasoning}")

In [ ]:

# Show auto-fail status
if mock_result.attributes_without_excerpts:
    print(f"\n⚠️  AUTO-FAIL: Missing excerpts for {', '.join(mock_result.attributes_without_excerpts)}")
else:
    print("\n✓ All attributes have supporting excerpts")

# Show auto-fail status if mock_result.attributes_without_excerpts: print(f"\n⚠️ AUTO-FAIL: Missing excerpts for {', '.join(mock_result.attributes_without_excerpts)}") else: print("\n✓ All attributes have supporting excerpts")

Excerpt Structure¶

Each excerpt includes:

{
    "text": str,              # Verbatim quote from response (empty if missing)
    "confidence": str,        # "low" | "medium" | "high" | "none"
    "similarity_score": float, # 0.0-1.0 (fuzzy match validation score)
    "explanation": str        # Optional: why excerpt couldn't be found
}

In [ ]:

# Example with excerpts
example_with_excerpts = {
    "drug_target": [
        {"text": "targets BCL-2 protein", "confidence": "high", "similarity_score": 0.95},
        {"text": "inhibits BCL-2", "confidence": "medium", "similarity_score": 0.87},
    ]
}

print("Example with excerpts:")
for attr, excerpts in example_with_excerpts.items():
    print(f"\n{attr}:")
    for exc in excerpts:
        print(f'  - "{exc["text"]}" (confidence: {exc["confidence"]}, similarity: {exc["similarity_score"]})')

# Example with excerpts example_with_excerpts = { "drug_target": [ {"text": "targets BCL-2 protein", "confidence": "high", "similarity_score": 0.95}, {"text": "inhibits BCL-2", "confidence": "medium", "similarity_score": 0.87}, ] } print("Example with excerpts:") for attr, excerpts in example_with_excerpts.items(): print(f"\n{attr}:") for exc in excerpts: print(f' - "{exc["text"]}" (confidence: {exc["confidence"]}, similarity: {exc["similarity_score"]})')

Use Cases¶

When to Use Deep-Judgment¶

✅ High-stakes evaluation where evidence transparency is critical:

• Medical diagnosis benchmarks
• Legal document analysis
• Scientific fact-checking
• Regulatory compliance

✅ Debugging parsing failures:

• Understanding why verification fails
• Identifying gaps in LLM responses
• Refining question or template design

✅ Quality assurance:

• Ensuring responses contain sufficient evidence
• Validating that answers aren't just plausible-sounding
• Auditing LLM reasoning processes

When NOT to Use Deep-Judgment¶

❌ High-volume verification where speed is critical:

• Deep-judgment is 3-5x slower than standard parsing
• Uses 3-5 LLM calls per question vs. 1 call for standard

❌ Low-stakes evaluation where audit trails aren't needed:

• Quick prototyping
• Informal testing
• Cost-sensitive applications

Performance Considerations¶

Execution Time¶

Deep-judgment significantly increases verification time:

• Standard parsing: 1 LLM call per question (~500-2000ms)
• Deep-judgment parsing: 3-5 LLM calls per question (~1500-10000ms)
  • Stage 1 (excerpts): 1 call + retries
  • Stage 2 (reasoning): 1 call
  • Stage 3 (parameters): 1 call

Impact: 3-5x slower than standard verification

Configuration Tips¶

Fuzzy Match Threshold¶

Controls how strictly excerpts must match the original response:

# Lenient matching (accepts paraphrases)
deep_judgment_fuzzy_match_threshold=0.70

# Default matching (balanced)
deep_judgment_fuzzy_match_threshold=0.80

# Strict matching (only very close matches)
deep_judgment_fuzzy_match_threshold=0.90

Trade-offs:

• Lower threshold (0.60-0.75): More lenient, may accept paraphrased excerpts
• Higher threshold (0.85-0.95): Stricter, only accepts near-exact matches

Best Practices¶

1. Start with Standard Parsing¶

Begin with standard parsing for your entire benchmark. Only enable deep-judgment when you need to:

• Debug specific parsing failures
• Audit high-stakes results
• Understand model behavior

2. Use Clear, Evidence-Based Templates¶

Design templates that expect explicit evidence in responses.

Related Features¶

Deep-judgment works alongside other advanced features:

• Abstention Detection: Detects when LLMs refuse to answer. Takes priority over deep-judgment auto-fail.
• Rubrics: Assess qualitative aspects. Use together with deep-judgment for comprehensive evaluation.
• Verification: Core verification system. Deep-judgment enhances standard verification with evidence extraction.