Dynamic Heuristic Shaping Experiment

Blind multi-model panel experiment showing heuristic shaping beats a deterministic control on 10 of 11 qualitative research-synthesis questions, with LLM-assisted shaping reaching 1725 Elo.

publication research heuristic-algebra LLM evaluation elo research-agent
achievement Updated 2026-06-16

Dynamic Heuristic Shaping Experiment

Title: Dynamic Heuristic Shaping Improves Qualitative Research Synthesis Date: June 16, 2026

A controlled experiment testing whether heuristic shaping makes a research agent produce better qualitative syntheses. Eleven research-agent schedules were each run under three conditions — deterministic control, deterministic heuristic shaping, and heuristic shaping with LLM-assisted claim extraction — over an identical source set, fixed clock, and schedule config. Outputs were judged blindly by a multi-model LLM panel (xAI, Gemini, Codex, Anthropic) against a fixed research-quality rubric covering question fit, evidence structure, falsifiability, and decision usefulness.

Headline result: Heuristic shaping beat the control on 10 of 11 questions. LLM-assisted shaping (heuristic_llm) ranked first on 8 of 11 and reached a global Elo of 1725.3, versus 1541.5 for deterministic heuristic shaping and 1233.2 for control — a +492-point spread over baseline.

Design

Condition Heuristic researcher LLM extraction
control off disabled
heuristic shaping disabled
heuristic_llm shaping enabled (xAI → Gemini → Codex → Anthropic fallback)
  • Suite: 11 research-agent schedules (financial synthesis, EEG decryption, HA social trading, Kalman optimality, ML clustering, LLM time-series baselines, backtest scaffolding, stationarity, data feeds, synthetic data, state-space forecasting).
  • Judging: anonymized summaries, RANKING: A, B, C outputs, 2–3 successful judges per question. Consensus via Borda count; agreement via Kendall’s W; global strength via sequential Elo (K=32, init 1500).
  • Judged artifact: domain_writer/summary.md per run.

Results

Condition Elo W–L Δ vs control
heuristic_llm 1725.3 62–18 +492.1
heuristic 1541.5 48–32 +308.3
control 1233.2 10–70 baseline

Mean Kendall’s W was 0.73. The lone control win was llm_time_series_baseline, where the control path used a specialized financial time-series template.

Interpretation

Heuristic shaping improves qualitative research synthesis, and the deterministic heuristic condition’s strong showing (1541 Elo, 48–32) shows the gain is not merely an artifact of extra LLM calls — the shaping layer itself adds useful structure. LLM-assisted extraction adds further coverage when identifying claims, counterclaims, base rates, and decision-relevant implications. Specialized domain templates (e.g., the financial time-series baseline) should still be preserved where they encode domain structure.

This is a qualitative research benchmark, not a calibration result. Brier-scored, resolved-outcome backtests remain the evidence tier for predictive-accuracy claims. Caveats: partial judge panels (~2.5 of 4 succeeded per question), uneven agreement (Kalman W=0.11), and a small, internally generated suite.

See Also