smnth / ctg-mm-eval-tradeoff-plots★ Featured · worked examples

Multiple Myeloma Extraction

Produce a small set of genuinely different charts from evaluation_matrix.csv (12 agent+model combinations scored against a ClinicalTrials.gov extraction benchmark) that each answer a distinct question a user would ask when choosing between models: "which is…

agent claude-codemodel anthropic/claude-sonnet-5snapshot python312-uveval programmatic9 stepsv1.3.0

Deploy Multiple Myeloma Extraction to your jetty.io

One-click installs this runbook into a collection on your Jetty account. You can run it from the Spot dashboard, schedule it, or pipe inputs in via the API.

Deploy on jetty.io →View source

Run time8-12 mins

Headline outputcharts/recall_vs_field_accuracy.png · charts/recall_vs_runtime.png · charts/cost_per_found_study.png · charts/agent_reliability_comparison.png · charts/recall_per_minute.png

Runs on Jetty's managed sandbox. No setup. Free for your first 10 runs.

Worked examples · 3

Real runs, real outputs.

visualizeeval ✓

A different 12-combination matrix — ulcerative-colitis benchmark (inset zoom)

Same runbook, a different evaluation matrix: 6 models x 2 agents scored against a 21-study ulcerative-colitis benchmark, where eight strong runs pile into the 95–100%…

recall_vs_field_accuracy.pngrecall_vs_runtime.pngcost_per_found_study.pngagent_reliability_comparison.pngrecall_per_minute.png

anthropic/claude-sonnet-5 · 11 minView run →Source

visualizeeval ✓

Same 12-combination matrix, re-run on claude-sonnet-5 via OpenRouter

Reproducibility check against the current runbook (v1.2.1, portable OpenRouter routing): all five charts rendered clean on the first attempt, the in-code zero-overlap…

recall_vs_field_accuracy.pngrecall_vs_runtime.pngcost_per_found_study.pngagent_reliability_comparison.pngrecall_per_minute.png

anthropic/claude-sonnet-5 · 7 minView run →Source

visualizeeval ✓

12 agent+model combinations, MM extraction benchmark

All 5 charts rendered clean — every one of the 12 combinations individually labeled with zero overlapping labels, palette and chart-form rules followed exactly.

recall_vs_field_accuracy.pngrecall_vs_runtime.pngcost_per_found_study.pngagent_reliability_comparison.pngrecall_per_minute.png

claude-sonnet-4-6 · 10 minView run →Source

The shape of the run

9 steps · start to finish.

Step 1

Environment Setup

▶

pip install pandas matplotlib adjustText
mkdir -p {{results_dir}}/charts

2
Step 2
Design System — Use These Exact Values (Do Not Substitute)
▶
This is a validated, colorblind-safe palette. Use it exactly — do not pick your own colors, do not use a matplotlib default cycle, do not use a rainbow/jet colormap.
3
Step 3
Compute Derived Data
▶
Write and run a Python script that:
4
Step 4
Render the 5 Charts
▶
Build each chart from Step 3's computed data using matplotlib and the exact palette from Step 2. For recall_vs_field_accuracy.png and recall_vs_runtime.png, color each point by its agent value using…
5
Step 5
Evaluate Outputs
▶
Status · Criteria PASS · All 5 PNGs exist, are non-trivial in size (>15KB), use the specified palette, have a legend where 2+ series are present, no chart uses a dual y-axis or 3D/pie form, every…
6
Step 6
Iterate on Errors (max 3 rounds)
▶
If any chart fails Step 5's criteria, fix the specific issue (palette substitution, missing legend, dual-axis violation, blank render, missing combination, overlapping label) and re-run. After each…
7
Step 7
Write Executive Summary
▶
Write {{results_dir}}/summary.md with one clearly-labeled section per chart (matching the 5 filenames), each containing: what the chart shows, and a one-sentence takeaway. Follow with a short "Which…
8
Step 8
Write Validation Report
▶
Write {{results_dir}}/validation_report.json:
9
Step 9
Final Checklist (MANDATORY — do not skip)
▶
echo "=== FINAL OUTPUT VERIFICATION ===" RESULTS_DIR="{{results_dir}}" for f in "$RESULTS_DIR/charts/recall_vs_field_accuracy.png" "$RESULTS_DIR/charts/recall_vs_runtime.png"…

Inputs

Evaluation matrixfilerequired

default: /app/assets/evaluation_matrix.csv

CSV, one row per agent+model combination, columns: agent, model, status, gold_total, num_found, num_missing, recall_pct, num_extra_entries, precision_pct, title_accuracy_pct, url_accuracy_pct, status_accuracy_pct, enrollment_accuracy_pct, study_type_accuracy_pct, start_date_accuracy_pct, overall_field_accuracy_pct, cost_usd, total_tokens, duration_seconds, notes.

Dependencies

pandas · required · Python package
matplotlib · required · Python package
adjustText · required · Python package

Required outputs

{{results_dir}}/charts/recall_vs_field_accuracy.png
Scatter, **all 12 runs**: x=recall_pct, y=overall_field_accuracy_pct. The 4 zero_output runs plot at (0, 0) using the critical-red failure marker (field accuracy is genuinely undefined when nothing was found, but 0,0 is the correct visual home for "found nothing, scored nothing" on this axis pair) — every one of the 4 must still get its own visible, non-overlapping label. The "which model is actually best" quadrant view.
{{results_dir}}/charts/recall_vs_runtime.png
Scatter, **all 12 runs**: x=duration_seconds (log scale), y=recall_pct. Failed/zero-output runs included as a distinct marker. The speed-vs-completeness trade-off.
{{results_dir}}/charts/cost_per_found_study.png
Bar chart, **all 12 runs** on the x-axis (one bar-slot per agent+model combination, same order as the other charts). The 4 runs with real `cost_usd` data get a real bar (y = cost_usd / num_found, dollars per correctly-found study). The 8 runs with no cost tracking (all opencode runs, plus any zero_output claude-code run lacking cost) get an explicit zero-height slot with an "no cost data" label sitting above the baseline — do NOT simply omit them from the chart, the point is every combination has a visible slot even when the honest answer is "unknown." Sort the 4 real bars ascending; place the "no data" slots after them, grouped and clearly separated.
{{results_dir}}/charts/agent_reliability_comparison.png
Grouped bar: claude-code vs opencode, 3 grouped metrics — success rate (scored / attempted for that agent), mean recall (scored runs only), mean field accuracy (scored runs only). The "which agent should I even use" view. (This chart is aggregate-level by design, not per-run — it does not need all 12 individual combinations, since the two agent bars already summarize all 13 attempts between them.)
{{results_dir}}/charts/recall_per_minute.png
Bar chart, **all 12 runs**: recall_pct / (duration_seconds / 60) — how fast each run reaches its recall level. Zero-output runs correctly compute to 0 (0% recall ÷ any runtime = 0) and appear as a visible zero-height bar with a label, not a gap. Sorted descending.
{{results_dir}}/chart_data.json
The exact derived numbers behind every chart (so results are auditable without re-deriving from the raw matrix)
{{results_dir}}/summary.md
One paragraph per chart: what it shows, and the one-sentence takeaway a user should draw from it
{{results_dir}}/validation_report.json
Structured validation results with stages, results, and overall_passed

Multiple Myeloma Extraction

Deploy Multiple Myeloma Extraction to your jetty.io

Real runs, real outputs.

A different 12-combination matrix — ulcerative-colitis benchmark (inset zoom)

Same 12-combination matrix, re-run on claude-sonnet-5 via OpenRouter

12 agent+model combinations, MM extraction benchmark

9 steps · start to finish.

Environment Setup

Design System — Use These Exact Values (Do Not Substitute)

Compute Derived Data

Render the 5 Charts

Evaluate Outputs

Iterate on Errors (max 3 rounds)

Write Executive Summary

Write Validation Report

Final Checklist (MANDATORY — do not skip)