Orchestration Lab — AI for Research

Five real social-science briefs, four ways to run a frontier model on them. The four arms are skills I wrote for my own work — three for Claude Code, one for the Codex CLI, all in the Open Science Skills toolkit. Every brief has a reference solution and a rubric written before any model ran. Pick a brief, pick an arm, paste one command, and grade what comes back against the answer key and our committed runs.

One draw. A captured run is one sample from a non-deterministic process, not a benchmark. Our numbers below are specimens. Re-run the briefs and expect yours to differ.

Read the report

Four ways to run a frontier model — why the leaderboards don't answer this, the full findings, what each setup caught and missed, and where it broke. On Pixels & Patterns.

Step 1Set up

You need Claude Code with the Open Science Skills plugin, R with the data packages, and the repo itself. That assumes Node and git on your machine, plus a paid claude.ai login. The Codex arm additionally needs the Codex CLI with a ChatGPT login.

npm install -g @anthropic-ai/claude-code
claude plugin marketplace add scdenney/open-science-skills
claude plugin install oss@open-science-skills
Rscript -e 'install.packages(c("projoint","ivdoctr","AER","car","causaldata","MatchIt"))'
git clone https://github.com/scdenney/ai-for-research.git
cd ai-for-research/demos/orchestration-lab

# Codex arm only — the Codex CLI plus the toolkit's Codex-side skills:
npm install -g @openai/codex
git clone https://github.com/scdenney/open-science-skills.git ~/open-science-skills
mkdir -p ~/.agents/skills
for s in ~/open-science-skills/codex/*/; do ln -sfn "${s%/}" ~/.agents/skills/"$(basename "$s")"; done

Every brief uses public data that ships with an R package — nothing to download. The runs call hosted models and cost real money: in our captures, $0.72 to $7.08 per cell on the Claude arms. Keep ANTHROPIC_API_KEY unset so headless runs bill your claude.ai plan, not an API account.

Step 2Pick a brief

The five briefs climb from a descriptive check to a real methods dispute. Each links to the exact prompt the models get.

easyDescribe — document a community-choice conjoint design (projoint's exampleData1) and check randomization balance.
standardEstimate — fit the AMCEs, cluster correctly, plot them with reference levels at zero.
moderateReviewer reply — answer a referee who calls the headline crime finding an artifact of the AMCE baseline.
hardIV replication — reproduce Acemoglu-Johnson-Robinson's colonial-origins 2SLS result (via ivdoctr) and stress it until the instrument breaks.
very hardMethods dispute — adjudicate Dehejia-Wahba vs Smith-Todd on whether matching recovers the LaLonde experimental benchmark (via causaldata).

Step 3Pick an arm and run it

Choose an arm and a brief. The command below updates; it writes into a fresh myruns/ leaf so the committed runs stay untouched. Full protocol, including the excerpt and hashing rules we used for the captured runs: run.md.

Arm Brief

What you'll get. The run writes its deliverables into your myruns/ leaf — typically script.R, a figure or two, and the brief's memo or summary — plus, on the headless Claude arms, claude-envelope.json, whose total_cost_usd and duration_ms fields are your cost and wall-clock. Grade the deliverables against the checklist below, then compare your cell against the committed run of the same cell under runs/.

Step 4Score your run

Every brief is graded on six yes-or-no items: four core facts a competent run must get right, one judgment call the answer key hands to the analyst, and one completeness credit for work beyond a correct answer. Pass means all four core items. Pass+ adds the judgment. Distinction means all six. Miss any core item and the run fails, whatever else it does well. Definitions and every captured run's item-by-item score: SCORING.md; executed reference solutions: reference/.

Describe — checklist

coreStates 400 respondents, 8 tasks each, 2 profiles per task, 6,400 profile rows.
coreNames all 7 attributes in readable form, not att1..att7.
corePer-attribute level counts read 3, 3, 4, 2, 4, 6, 2.
coreIdentifies the repeated, flipped task 1 and does not count it as a 9th task.
judgmentFlags Total Daily Driving Time as the lone imbalance; makes no "perfect balance" claim.
completenessReports the exact max deviation from uniform (~1.94 pp), not only a min-max spread.

Estimate — checklist

coreViolent Crime Rate is the largest |AMCE|, magnitude right for its stated scale (~25.1 pp corrected / ~16.5 uncorrected, never mislabeled).
coreEvery large effect signs correctly; attribute ordering matches the key.
coreStandard errors clustered on respondent id.
coreEvery attribute's reference level fixed at zero; estimates presented as AMCEs.
judgmentStates corrected or uncorrected as a deliberate, labeled choice.
completenessNames the profile-level estimand and the IRR mechanism (tau ~0.17, ×1.52), explained not asserted.

Reviewer reply — checklist

coreConcedes multi-level AMCEs shift under relabeling, backed by a number from these data.
coreStates crime is binary, so a baseline flip only flips the sign — |AMCE| invariant.
coreComputes marginal means (.626 / .374) and uses the baseline-free MM range as the ranking currency.
coreCaps the claim: crime is the largest single driver, commute-comparable — never "dominates."
judgmentFlags the ~1.4 pp crime-vs-commute gap as within noise — a statistical tie.
completenessReports both magnitudes, uncorrected 16.5 and corrected 25.1 pp.

IV replication — checklist

coreReproduces the headline: 2SLS 0.944, OLS 0.522, first-stage F 22.95.
coreRuns the stress specs: controls keep F > 10; dropping the neo-Europes pushes F to 8.65; Africa-only collapses to F = 0.30.
coreFlags the weak-instrument specs as weak.
coreNo overclaim about what the replication proves.
judgmentStates the two-sided ceiling: a collapsed first stage neither confirms nor overturns the original result.
completenessOne unified table across all five specifications.

Methods dispute — checklist

coreAnchors exact: experimental benchmark +$1,794; naive CPS comparison −$8,498.
coreRuns the specification curve including pre-earnings matching (demographics-only must fail; pre-earnings 1-NN lands near the benchmark).
coreBenchmark-referenced results table.
coreNo overclaim in the adjudication.
judgmentLands the "helps but does not settle" verdict — favorable-specification recovery, not universal recovery.
completenessBenchmark-referenced figure.

What our runs showed

Twenty captured cells, graded blind against the rubrics, and one rule sums them up. Difficulty picks the setup, not effort. The full findings and cost breakdowns are in the report. The run-by-run matrix is in RESULTS.md.

Four small panels, one per arm, each plotting rubric items met across the five briefs against dotted threshold lines labeled Pass, Pass plus, and Distinction. — Rubric items met across the five briefs, one panel per arm, read against the same three band thresholds. Our captures: one draw each.

Setup	When to reach for it
Advisor	The default for a single, well-scoped analysis. The only arm that took the judgment-heavy reviewer reply to the top band, because its second read re-computes the work and tests the judgment call formally rather than eyeballing it. Reach for it first.
Fable lead	When the task is too big to hold in one context — a broad migration, a multi-file audit, a many-source survey. That regime is untested here. The case for it is the context economics. On these briefs it clears every core item at the lowest Claude-orchestrator cost.
Opus lead	Rarely, on work this size. It ties the Fable lead on the two hardest briefs at higher cost, and its edge on the easier briefs is write-up thoroughness — persistent across its captures, but completeness rather than correctness. Reserve it for a task hard enough to strain a top model.
Codex lead	A cross-vendor read on the same brief. The Sol lead matched the Fable lead's total (25 of 30 items) and took both hard briefs to Distinction, at roughly the Fable arm's cost at list prices. Reach for it when you want the second opinion to come from a different vendor entirely.

Our captures are one draw each. Re-run a cell and expect yours to differ.