karenina.benchmark.verification.extension

extension

Extend a prior verification run along three axes: judges, answerers, replicates.

Mirrors the error-analysis facade pattern: result-set-in, result-set-out. A ReplayStore built from the prior results serves the answering stage for triples already covered; new answerers or replicates miss and run live. VerificationConfig.skip_triples tells the batch runner to drop tasks already present in prior_results so those rows pass through verbatim.

This module also hosts :func:extend_rubric_run, a sibling facade for attaching a new rubric to a prior run. Unlike the judge/answerer/replicate extension, rubric extension enriches prior rows in-place: answering is replayed, template parsing and verification are skipped (evaluation_mode="rubric_only"), and the resulting rubric scores are merged trait-by-trait onto the prior rows. Shape and row count are preserved; same-name trait collisions raise.

Classes

Functions

extend_rubric_run

extend_rubric_run(
    benchmark: Benchmark,
    prior_results: VerificationResultSet,
    config: VerificationConfig,
    *,
    run_name: str | None = None,
    question_ids: list[str] | None = None,
    async_enabled: bool | None = None,
    progress_callback: Callable[[float, str], None]
    | None = None,
    sink: Any = None,
) -> VerificationResultSet

Attach a new rubric to an existing verification run.

The caller sets the new rubric on benchmark (global and/or per-question) before invoking this helper. A replay store built from prior_results serves the answering stage, the pipeline runs in evaluation_mode="rubric_only" (template verification skipped), and the resulting rubric scores are merged trait-by-trait onto copies of the prior rows. Row count is preserved: the merged set has exactly len(prior_results.results) entries.

Parameters:

Name	Type	Description	Default
benchmark	Benchmark	The :class:Benchmark carrying the new rubric(s).	required
prior_results	VerificationResultSet	Result set from an earlier run_verification call. Must be non-empty.	required
config	VerificationConfig	Verification configuration describing the same shape as prior_results (answering_models, parsing_models, replicate_count must equal the observed prior shape). replay_store must be None; the helper owns that slot. evaluation_mode must be either the default "template_only" or "rubric_only" (the helper rewrites it internally).	required
run_name	str | None	Optional override for the merged run name. Defaults to the run name carried by prior_results.	None
question_ids	list[str] | None	Optional subset of question IDs. Defaults to every question present in prior_results.	None
async_enabled	bool | None	Forwarded to run_verification.	None
progress_callback	Callable[[float, str], None] | None	Forwarded to run_verification.	None
sink	Any	Optional :class:~karenina.benchmark.verification.sinks.ResultSink forwarded to run_verification. The sink receives the rubric_only rows produced by the extension pipeline (one per prior triple); it never sees the final enriched rows returned to the caller. Unlike extend_template, prior rows are not seeded into the sink because rubric-only rows reuse the same task keys; seeding would duplicate entries in append-only sinks. Pair with a :class:~karenina.benchmark.verification.sinks.ProgressiveFileSink to make the rubric-only pass resumable; persisting the merged enriched output is the caller's responsibility.	None

Returns:

Type	Description
VerificationResultSet	VerificationResultSet of enriched prior rows (same count as
VerificationResultSet	prior_results.results) carrying the new rubric scores.

Raises:

Type	Description
ValueError	On any input validation failure (empty prior set, shape mismatch, pre-populated replay_store, unsupported evaluation_mode, no rubric attached, metric traits present, inconsistent run names without override, trait name collision, or shape corruption at merge time).

Source code in src/karenina/benchmark/verification/extension.py

def extend_rubric_run(
    benchmark: Benchmark,
    prior_results: VerificationResultSet,
    config: VerificationConfig,
    *,
    run_name: str | None = None,
    question_ids: list[str] | None = None,
    async_enabled: bool | None = None,
    progress_callback: Callable[[float, str], None] | None = None,
    sink: Any = None,
) -> VerificationResultSet:
    """Attach a new rubric to an existing verification run.

    The caller sets the new rubric on ``benchmark`` (global and/or
    per-question) *before* invoking this helper. A replay store built
    from ``prior_results`` serves the answering stage, the pipeline
    runs in ``evaluation_mode="rubric_only"`` (template verification
    skipped), and the resulting rubric scores are merged
    trait-by-trait onto copies of the prior rows. Row count is
    preserved: the merged set has exactly ``len(prior_results.results)``
    entries.

    Args:
        benchmark: The :class:`Benchmark` carrying the new rubric(s).
        prior_results: Result set from an earlier ``run_verification``
            call. Must be non-empty.
        config: Verification configuration describing the **same shape**
            as ``prior_results`` (``answering_models``,
            ``parsing_models``, ``replicate_count`` must equal the
            observed prior shape). ``replay_store`` must be ``None``;
            the helper owns that slot. ``evaluation_mode`` must be
            either the default ``"template_only"`` or ``"rubric_only"``
            (the helper rewrites it internally).
        run_name: Optional override for the merged run name. Defaults
            to the run name carried by ``prior_results``.
        question_ids: Optional subset of question IDs. Defaults to every
            question present in ``prior_results``.
        async_enabled: Forwarded to ``run_verification``.
        progress_callback: Forwarded to ``run_verification``.
        sink: Optional :class:`~karenina.benchmark.verification.sinks.ResultSink`
            forwarded to ``run_verification``. The sink receives the
            ``rubric_only`` rows produced by the extension pipeline (one per
            prior triple); it never sees the final enriched rows returned to
            the caller. Unlike ``extend_template``, prior rows are **not**
            seeded into the sink because rubric-only rows reuse the same
            task keys; seeding would duplicate entries in append-only sinks.
            Pair with a
            :class:`~karenina.benchmark.verification.sinks.ProgressiveFileSink`
            to make the rubric-only pass resumable; persisting the merged
            enriched output is the caller's responsibility.

    Returns:
        ``VerificationResultSet`` of enriched prior rows (same count as
        ``prior_results.results``) carrying the new rubric scores.

    Raises:
        ValueError: On any input validation failure (empty prior set,
            shape mismatch, pre-populated ``replay_store``, unsupported
            ``evaluation_mode``, no rubric attached, metric traits
            present, inconsistent run names without override, trait
            name collision, or shape corruption at merge time).
    """
    _validate_rubric_extension(benchmark, prior_results, config, run_name)

    effective_run_name = run_name or _infer_run_name(prior_results)
    effective_question_ids = (
        list(question_ids)
        if question_ids is not None
        else sorted({r.metadata.question_id for r in prior_results.results})
    )

    replay_store = capture_from_result_set(
        prior_results,
        include_parsed=False,
        include_agent_traces=True,
        only_successful=False,
        replicate_selector="all",
    )
    logger.info(
        "extend_rubric: captured replay store with %d entries from %d prior results",
        len(replay_store.entries),
        len(prior_results.results),
    )

    extended_config = config.model_copy(update={"replay_store": replay_store, "evaluation_mode": "rubric_only"})

    # Note: no pre-seeding here. Rubric-only rows share task keys with prior
    # rows, so seeding would produce duplicate append-only entries in the
    # sink (one prior row, one rubric-only row per key). The sink is
    # durability for the rubric-only pass; the enriched merged set is the
    # return value and the caller's responsibility to persist.

    new_results = benchmark.run_verification(
        config=extended_config,
        question_ids=effective_question_ids,
        run_name=effective_run_name,
        async_enabled=async_enabled,
        progress_callback=progress_callback,
        sink=sink,
    )

    enriched = _enrich_with_rubric(prior_results, new_results, effective_run_name)
    logger.info(
        "extend_rubric: enriched %d prior rows under run_name=%r",
        len(enriched.results),
        effective_run_name,
    )
    return enriched

extend_template_run

extend_template_run(
    benchmark: Benchmark,
    prior_results: VerificationResultSet,
    config: VerificationConfig,
    *,
    run_name: str | None = None,
    question_ids: list[str] | None = None,
    async_enabled: bool | None = None,
    progress_callback: Callable[[float, str], None]
    | None = None,
    sink: Any = None,
) -> VerificationResultSet

Extend a prior verification run along any combination of three axes.

The axes, all optional and composable in a single call:

1. Parsing models (judges): new judge(s) in config.parsing_models.
2. Answering models: new answerer(s) in config.answering_models.
3. Replicates: higher config.replicate_count than the prior fan-out.

A :class:~karenina.replay.ReplayStore is built from prior_results so that prior (question, answerer, replicate) combinations serve the answering stage from replay. New answerers, new replicates, or any combination thereof miss the store and run answering live. Parsing always runs live. Triples already covered by prior_results are filtered out of the task queue so prior rows pass through the merge verbatim. The returned set matches the shape of a joint run with the full (answerers × judges × replicates) matrix.

Parameters:

Name	Type	Description	Default
benchmark	Benchmark	The :class:Benchmark whose questions and templates the prior run evaluated. Question IDs on prior_results must exist on this benchmark.	required
prior_results	VerificationResultSet	Result set from an earlier run_verification call to extend. Must be non-empty.	required
config	VerificationConfig	Verification configuration for the extension. Must describe the final state (full union, not deltas): parsing_models = every judge you want in the merged output (old judges from prior_results + any new judges). answering_models = every answerer you want in the merged output. Must be a superset of the answering identities in prior_results (compared via :meth:ModelIdentity.from_model_config). replicate_count = the final number of replicates. Must be >= the fan-out observed in prior_results. replay_store must be None (the extension owns that slot).	required
run_name	str | None	Optional override for the merged run name. Defaults to the run name carried by prior_results when all rows agree; raises ValueError on disagreement when no override is supplied.	None
question_ids	list[str] | None	Optional subset of question IDs to re-judge. Defaults to every question in prior_results.	None
async_enabled	bool | None	Forwarded to run_verification.	None
progress_callback	Callable[[float, str], None] | None	Forwarded to run_verification.	None
sink	Any	Optional :class:~karenina.benchmark.verification.sinks.ResultSink forwarded to run_verification so newly produced rows are persisted as they complete. Prior rows are pre-seeded into the sink via :meth:~karenina.benchmark.verification.sinks.ResultSink.seed_prior_results before the pipeline starts, so the sink's final export reflects the full merged shape (prior + new). Pair with a :class:~karenina.benchmark.verification.sinks.ProgressiveFileSink to make the extension resumable.	None

Returns:

Type	Description
VerificationResultSet	Merged VerificationResultSet with ``len(prior_results.results)
VerificationResultSet	len(new.results)rows, all stamped with the effectiverun_name``.

Raises:

Type	Description
ValueError	On any input validation failure (empty prior set, missing parsing models, answering identity mismatch, replicate count mismatch, pre-populated replay_store, or inconsistent run names without an override).

Source code in src/karenina/benchmark/verification/extension.py

def extend_template_run(
    benchmark: Benchmark,
    prior_results: VerificationResultSet,
    config: VerificationConfig,
    *,
    run_name: str | None = None,
    question_ids: list[str] | None = None,
    async_enabled: bool | None = None,
    progress_callback: Callable[[float, str], None] | None = None,
    sink: Any = None,
) -> VerificationResultSet:
    """Extend a prior verification run along any combination of three axes.

    The axes, all optional and composable in a single call:

    1. **Parsing models (judges)**: new judge(s) in ``config.parsing_models``.
    2. **Answering models**: new answerer(s) in ``config.answering_models``.
    3. **Replicates**: higher ``config.replicate_count`` than the prior fan-out.

    A :class:`~karenina.replay.ReplayStore` is built from ``prior_results`` so
    that prior ``(question, answerer, replicate)`` combinations serve the
    answering stage from replay. New answerers, new replicates, or any
    combination thereof miss the store and run answering live. Parsing
    always runs live. Triples already covered by ``prior_results`` are
    filtered out of the task queue so prior rows pass through the merge
    verbatim. The returned set matches the shape of a joint run with the
    full ``(answerers × judges × replicates)`` matrix.

    Args:
        benchmark: The :class:`Benchmark` whose questions and templates the
            prior run evaluated. Question IDs on ``prior_results`` must exist
            on this benchmark.
        prior_results: Result set from an earlier ``run_verification`` call
            to extend. Must be non-empty.
        config: Verification configuration for the extension. Must describe
            the **final** state (full union, not deltas):

            - ``parsing_models`` = every judge you want in the merged output
              (old judges from ``prior_results`` + any new judges).
            - ``answering_models`` = every answerer you want in the merged
              output. Must be a superset of the answering identities in
              ``prior_results`` (compared via
              :meth:`ModelIdentity.from_model_config`).
            - ``replicate_count`` = the final number of replicates. Must be
              ``>=`` the fan-out observed in ``prior_results``.
            - ``replay_store`` must be ``None`` (the extension owns that slot).
        run_name: Optional override for the merged run name. Defaults to the
            run name carried by ``prior_results`` when all rows agree; raises
            ``ValueError`` on disagreement when no override is supplied.
        question_ids: Optional subset of question IDs to re-judge. Defaults
            to every question in ``prior_results``.
        async_enabled: Forwarded to ``run_verification``.
        progress_callback: Forwarded to ``run_verification``.
        sink: Optional :class:`~karenina.benchmark.verification.sinks.ResultSink`
            forwarded to ``run_verification`` so newly produced rows are
            persisted as they complete. Prior rows are pre-seeded into the
            sink via :meth:`~karenina.benchmark.verification.sinks.ResultSink.seed_prior_results`
            before the pipeline starts, so the sink's final export reflects
            the full merged shape (prior + new). Pair with a
            :class:`~karenina.benchmark.verification.sinks.ProgressiveFileSink`
            to make the extension resumable.

    Returns:
        Merged ``VerificationResultSet`` with ``len(prior_results.results)
        + len(new.results)`` rows, all stamped with the effective ``run_name``.

    Raises:
        ValueError: On any input validation failure (empty prior set, missing
            parsing models, answering identity mismatch, replicate count
            mismatch, pre-populated ``replay_store``, or inconsistent run
            names without an override).
    """
    _validate(prior_results, config, run_name)

    effective_run_name = run_name or _infer_run_name(prior_results)
    effective_question_ids = (
        list(question_ids)
        if question_ids is not None
        else sorted({r.metadata.question_id for r in prior_results.results})
    )

    replay_store = capture_from_result_set(
        prior_results,
        include_parsed=False,
        include_agent_traces=True,
        only_successful=False,
        replicate_selector="all",
    )
    logger.info(
        "extend_template: captured replay store with %d entries from %d prior results",
        len(replay_store.entries),
        len(prior_results.results),
    )

    skip_triples = frozenset(
        (
            r.metadata.question_id,
            r.metadata.answering.canonical_key,
            r.metadata.parsing.canonical_key,
            r.metadata.replicate,
        )
        for r in prior_results.results
    )

    observed_replicates = _observed_replicate_count(prior_results)
    prior_answering_keys: set[str] = {r.metadata.answering.canonical_key for r in prior_results.results}
    new_answering_keys = sorted(
        {ModelIdentity.from_model_config(m, role="answering").canonical_key for m in config.answering_models}
        - prior_answering_keys
    )
    added_replicates = config.replicate_count - observed_replicates
    logger.info(
        "extend_template: new answerers=%s, added replicates=%d, skip_triples=%d",
        new_answering_keys or "[]",
        added_replicates,
        len(skip_triples),
    )

    extended_config = config.model_copy(update={"replay_store": replay_store, "skip_triples": skip_triples})

    _seed_sink_if_supported(sink, prior_results)

    new_results = benchmark.run_verification(
        config=extended_config,
        question_ids=effective_question_ids,
        run_name=effective_run_name,
        async_enabled=async_enabled,
        progress_callback=progress_callback,
        sink=sink,
    )

    merged = _merge(prior_results, new_results, effective_run_name)
    logger.info(
        "extend_template: merged %d prior + %d new = %d total results under run_name=%r",
        len(prior_results.results),
        len(new_results.results),
        len(merged.results),
        effective_run_name,
    )
    return merged