Insights DCLASS: Pipedrive schema qualification

Overall frozen acceptance: **FAIL**. Target real correctness changes by 0.00 questions per four-question repeat (0.00 percentage points), with paired family-cluster 95% interval [0.00, 0.00] questions. The control observed repeat range is 0.00 questions; the candidate range is 0.00. 0 of three candidate repeats exceed the control mean.

Target real silent-wrong answers change from 0.67 to 2.33 per repeat: difference 1.67 [0.67, 2.67] questions, against a control range of 1.00. The whole-sample and complementary-set safety gates are aggregate tests; their PASS labels do not establish target safety or outweigh failed real-question correctness. The target tables below must be read alongside those aggregate gates.

The production change preserves the Pipedrive schema identifier at the shared generation-context loader. It changes no generator, verifier, repair policy, oracle, source allowlist or business predicate. The observed-defect target was frozen from D3 records: four real first-turn questions across P02/P03/P05/X01 and sixteen fresh cases across P01–P05. Broader family populations are reported separately.

This is a development-informed comparison against historical fresh controls and mixed-date real controls. The target was selected from historical failures, so regression to the mean is possible. The family intervals condition on the observed repeats and do not model provider or run-level uncertainty. Every latency below was measured on a shared host; historical and new timing conditions differ.

Per-repeat outcomes

First-turn errors use supported first turns. Total errors include task and infrastructure failures on every turn. Refused combines unverified and justified refusals; the exhaustive outcome and guard partitions follow. Precision excludes comparison-unverified answers. Counts, means and ranges describe completed evaluator attempts.

Repeat means and ranges

Cells are mean [minimum, maximum] over defined repeats; precision also states its defined-repeat n. Count summaries use all three repeats. Target-control precision is undefined in repeat 1 for both datasets, so those two means use n=2; undefined precision is not replaced by zero. The control max−min range is the acceptance noise floor, an observed range rather than a confidence bound. Candidate ranges are also shown in the noise table.

Outcome and refusal partitions

Each outcome partition sums to that repeat’s attempt denominator; each guard partition sums to its refusal count. No error, blocked turn, clarification or unverified result is silently counted as a refusal or a correct answer.

Paired intervals and observed noise

Candidate minus control; count differences are normalized to one repeat of the named set. 10,000 whole-family resamples, seed 2026090611, nearest-rank 2.5th and 97.5th percentiles. Each arm contributes its repeat-mean family counts; repeat indices are not causal pairs. Precision and p95 are pooled within the bootstrap and may differ from differences of repeat means.

Frozen pass targets

The target real correct-answer gain is inside or at the observed control noise floor. The frozen target correctness requirement therefore fails, regardless of whether the schema bug is removed.

A safety PASS means the frozen adverse-difference tests did not trigger; it is not proof of equivalence. Whole fresh and whole real apply the same safety checks as both complements. The target real conjunction also requires two improved candidate repeats. Its four family clusters provide limited resolution: gains concentrated in only one or two families may not clear the interval.

Affected-family before and after

The first-turn metric counts distinct first-turn cases and excludes derivative turns. All-attempt completion is shown separately. Cells are means [minimum, maximum], then a fixed denominator. A family absent from a set has denominator zero and an undefined rate.

Candidate target defects remaining

The unchanged D3 structural classifier was applied after measurement. Classes overlap; distinct cases are counted once per class across repeats, and pooled attempts count each occurrence. Refusals and comparison-unverified answers have no inferred semantic defect class and remain separately accounted. A generic row mismatch does not isolate its cause.

fresh target outcome accounting: `{"comparison_unverified": {"distinct_cases": 10, "pooled_attempts": 22}, "correct_completed": {"distinct_cases": 2, "pooled_attempts": 5}, "refusal_unverified": {"distinct_cases": 4, "pooled_attempts": 8}, "silent_wrong": {"distinct_cases": 4, "pooled_attempts": 7}, "task_error": {"distinct_cases": 5, "pooled_attempts": 6}}`. Unclassified wrong/error attempts: 0.

real target outcome accounting: `{"comparison_unverified": {"distinct_cases": 1, "pooled_attempts": 3}, "silent_wrong": {"distinct_cases": 3, "pooled_attempts": 7}, "task_error": {"distinct_cases": 2, "pooled_attempts": 2}}`. Unclassified wrong/error attempts: 0.

Outcome transitions against control

Every matched-case Cartesian control-repeat × candidate-repeat pair has equal weight (nine pairings). Counts sum to one repeat’s denominator for the named set. These are outcome movements, not proven corrections.

Provider identity and provenance

Requested zai model: glm-5.2; temperature 0, seed 42, thinking disabled, timeout 25 seconds. Defaults remain in generation; verifier and repair are off. This table covers generator dispatches only. Returned response.model values are retained for completed generator calls and linked to attempt job IDs. They are provider metadata, not independent serving attestation. Missing links stay explicit. Failed calls and calls without terminal records do not attest a served model.

A separate auxiliary wrapper retained 596 completed invocations. Its model/provider labels come from execution configuration, not returned response identity. Returned auxiliary identities and individual internal failed/fallback dispatches were not separately retained and remain unverified. These records cannot establish a complete provider-dispatch inventory. `auxiliary-provenance.json` records per-run counts and source hashes; `auxiliary/` retains the structural call/job linkage with explicitly named configured-model fields. No missing calls or identities were reconstructed.

New-run chronology: queue-to-entry includes process launch and scheduling, so it is an upper bound on lock wait. Start below is driver entry, before host admission; finish includes owned-process cleanup.

Historical D3 fresh control repeats and real control repeat 1 used revision 2513a718. Two new real control repeats use the D3 tip; its documented changes are obligation-recording and sampling-label telemetry. The D3 offline replay reproduced outcomes but did not remeasure telemetry cost. The three candidate repeats per dataset use one frozen revision. Corrected corpus and evaluator hashes agree within each dataset, as do requested generator/verifier configurations. DCLASS containment and telemetry wrappers differ from historical D3; their separate source hashes are retained. The DCLASS harness sets `LORE_GRANOLA_INDEX_AUTO_INIT=0` to prevent shared-index initialization writes. This containment setting differs from historical wrapper behavior; it is explicitly retained as a method difference. Fresh-v1 was already exposed in D2/D3, and historical failures plus question shapes informed this root fix. There is no claim of an unseen holdout or a sole-cause production effect.

New measurements hold `/tmp/lore-eval.lock` across admission, run and cleanup, use two workers and application ports 8700–8799, and retain queue-to-entry wait evidence. This does not eliminate unrelated shared-host activity. Frozen application time is 2026-09-06 12:00 UTC, with Europe/London business periods. Per-question p95 includes every attempt with a retained feature-completion time; failed-call timings remain in traces.

The session resumed after five new runs had completed, retaining all three candidate fresh repeats and both supplemental real controls. The final fresh receipt records successful completion and cleanup although its controller sequence event was not written. Receipt validation allowed the three remaining real repeats to proceed without rerunning any completed measurement. The interruption adds a substantial time gap before candidate real measurements; provider and host conditions across that gap are not controlled. `resume-receipt.json` preserves the recovery evidence.

Checks, limitations and review

Check evidence: `{"curated": {"errors": 0, "failed": 0, "passed": 636, "skipped": 88}, "errors": 0, "exit_code": 0, "failed": 0, "files": 46, "harness": {"errors": 0, "failed": 0, "passed": 43, "receipt": "checks/harness-final-checks.json", "result_line": "43 passed in 13.17s", "ruff": "All checks passed!"}, "missing_inherited": ["tests/js/bi-terminal-rendering.test.mjs", "tests/test_answer_contract_role_promotion.py", "tests/test_federated_semantic_refusal.py", "tests/test_insights_cold_schema_context.py", "tests/test_insights_federated_followup_contract.py", "tests/test_insights_n01_definition_schema.py", "tests/test_insights_v16_aggregate_group_scope.py", "tests/test_insights_v16_rank_words.py", "tests/test_insights_v17_distinct_snapshot_sum.py", "tests/test_insights_v17_explicit_output.py", "tests/test_insights_v17_membership_predicates.py", "tests/test_insights_v17_output_recovery_native.py", "tests/test_insights_v18_membership_output.py", "tests/test_insights_v18_money_guard_recovery.py", "tests/test_insights_v8_chain_recovery.py", "tests/test_insights_v8_context_contract.py", "tests/test_insights_v8_ct_input_regressions.py", "tests/test_insights_v8_intent_shape_diagnostics.py", "tests/test_insights_v9_contract_review.py"], "missing_inherited_files": 19, "passed": 1045, "receipt_tests": {"errors": 0, "failed": 0, "passed": 8, "result_line": "8 passed in 13.14s", "ruff": "All checks passed!"}, "result_lines": ["1045 passed, 92 skipped, 23 warnings in 191.92s (0:03:11)"], "skipped": 92, "wall_seconds": 202.26604114286602}`.

Zero observed curated violations covers present executed tests only. Skipped and absent inherited tests are unverified. The ratified staff/house and Pipedrive-only identity entries are applied unchanged where relevant; owner dependencies remain pending. In particular X01 people comparisons and P02 listings can depend on these meanings. The benchmark uses real question shapes against a fixture and corrected oracle; it does not establish correctness on live customer data. Family-bootstrap intervals contain only four target real clusters and omit run-level uncertainty. The real-target correctness interval [0,0] is degenerate conditional on the observed all-zero counts; it does not establish population equivalence or serving stability. Each four-question real-target nearest-rank p95 equals that repeat’s maximum, rather than a well-estimated population tail.

Independent review and correction ledger

Three separate foreground Codex CLI processes reviewed checkpoint `5e7e25b6` with requested model `gpt-6-astra`, reasoning `high`, full-access sandbox and strict no-modification instructions. Each exited successfully and detected zero changed inputs among 6,507 checked files. The additional inventory covering 568 report, PR, source, corpus and full/compact measurement inputs also detected zero changes across all three reviews. Prompts, replies and receipts are retained; the CLI banner is configuration evidence, not independent serving attestation.

The reviewed results SHA-256 is `1d79d83fb1a8fad395729d303a95003944e361650c01526fbf8ccc4beba58a11`. All quality metrics, intervals, transitions, thresholds, membership, raw records and application code remain unchanged after review. Corrections affect reporting and structural provenance only; no missing auxiliary calls or model identities were reconstructed and no measurement was rerun. These reporting corrections were applied after the three reviews and checked locally; no fourth independent review is claimed.

Before those reviews, the resumed session also corrected the refusal-partition description, an environment-setting name and an inherited generator model-source label; added run chronology and the interrupted-session disclosure; and foregrounded the target wrong-answer increase. The complete compact JSONL records are now committed despite the repository-wide JSONL ignore rule. These were reporting/retention corrections, not application tuning.

Independent reviews validated structural outcome fields, timing and retained checks. They did not perform new semantic answer scoring, execute the application tests, verify live serving identity or validate sibling integration. The absence of a classified schema-qualification error does not establish complete semantic diagnosis or correctness of unverified answers.

Expected merge order: D1+D2+D3 (PR #104), then DCLASS, then sibling F01. The localized schema-loader correction establishes the shared boundary before broader family-context work; F01 must preserve it when resolving overlap. Combined sibling behavior is unverified. No application merge or deployment is part of this report.

Full retained records: `/tmp/lore-goal3-eval/goal-dclass/`; historical controls: `/tmp/lore-goal3-eval/goal-d3/`. Compact records: `docs/evidence/insights-goal-dclass-2026-09-12/`. Recompute with `PYTHONDONTWRITEBYTECODE=1 /home/stevan/dev/lore/.venv/bin/python scripts/insights_dclass_report.py --compact`. Compact evidence excludes question text, SQL, rows and free-text reasoning. HTML publication is public: random naming and noindex discourage discovery and indexing but do not provide access control.