Results
Three-way: spp vs EvoPrompt vs DSPy, on gpt-5-nano.
All three arms ran on the same task model — gpt-5-nano (reasoning_effort=low, temperature=1.0) — over the same sacred test rows (AG News 1,000 · SST-5 1,000 · TREC 500), scored with the same label-matching wrapper, and started from the identical seed prompt. The honest comparator is dollars: gpt-5-nano lists at $0.05 / 1M input and $0.40 / 1M output, so output costs 8× input and raw token counts mislead.
Regime note — read this first; it matters for fairness. EvoPrompt and spp ran 0-shot (instruction only). DSPy (MIPROv2) ran few-shot — it bootstraps up to four in-context demonstrations, because that is DSPy’s design strength. We ran it in its strongest honest configuration rather than hobble it to 0-shot. DSPy therefore carries extra per-row input tokens for its demos; that is a real, disclosed asymmetry, not a measurement artifact.
1. Headline: accuracy
Test accuracy on the sacred holdout. Bold = best on that task. SE ≈ 0.010 (1,000 rows) / 0.014 (500 rows).
| Task | Seed | EvoPrompt (0-shot) | spp (0-shot) | DSPy (few-shot) |
|---|---|---|---|---|
| AG News | 0.870 | 0.869 | 0.876 | 0.881 |
| SST-5 | 0.557 | 0.561 | 0.579 | 0.580 |
| TREC | 0.828 | 0.804 | 0.924 | 0.874 |
| Mean | 0.752 | 0.745 | 0.793 | 0.778 |
Reading it honestly, task by task:
- AG News — three-way statistical tie. DSPy 0.881, spp 0.876, EvoPrompt 0.869 all sit inside a single standard error (~0.010). The seed (0.870) was already near the dataset’s saturation ceiling, so nobody could move it much. No real accuracy winner here — the separation is entirely in cost (§2).
- SST-5 — spp and DSPy tie, both clearly beat the rest. spp 0.579 ≈ DSPy 0.580 (a 0.001 gap — noise). Both clear EvoPrompt (0.561) and the seed (0.557) by ~1.8–2.3 points (~2 SE). The notable result: spp’s 0-shot rule-writing matched DSPy’s few-shot accuracy with zero demonstrations.
- TREC — spp wins decisively, and it is not close. spp 0.924 beats DSPy 0.874 by 5.0 points (~3.5 SE on 500 rows — a real gap, not noise), beats the seed by 9.6, and beats EvoPrompt by 12.0. This is the one task with a clear accuracy winner, and it is spp.
EvoPrompt never beat its own seed on any task (it tied on AG News/SST-5 and regressed on TREC, 0.804 < 0.828). Its GA converged back to — or below — the human starting prompt every time on this model.
2. The deciding axis: cost (USD)
Total gpt-5-nano spend per arm per task (search/optimization + sacred-test scoring).
| Task | EvoPrompt | spp | DSPy |
|---|---|---|---|
| AG News | $0.18 | $0.04 | $0.15 |
| SST-5 | $0.21 | $0.10 | $0.19 |
| TREC | $0.24 | $0.11 | $0.18 |
| Total | $0.63 | $0.25 | $0.52 |
spp is the cheapest arm on every task — roughly 2.5× cheaper than DSPy and 2.5× cheaper than EvoPrompt overall, while posting the highest mean accuracy. On the two tasks where accuracy is a tie (AG News, SST-5), cost is the only thing that separates the arms, and spp wins it outright. On the task where accuracy is not a tie (TREC), spp wins that too and is cheapest.
3. Tokens — why raw count is the misleading metric
Total tokens tell a different (and unfair) story than dollars, because the three arms have opposite input/output profiles.
| Task | Arm | Calls | Input tok | Output tok | Total tok | Cost |
|---|---|---|---|---|---|---|
| AG News | EvoPrompt | 4,288 | 394,645 | 400,068 | 794,713 | $0.18 |
| spp | 1,643 | 430,852 | 43,060 | 473,912 | $0.04 | |
| DSPy | 2,071 | 1,327,021 | 205,039 | 1,532,060 | $0.15 | |
| SST-5 | EvoPrompt | 4,368 | 289,403 | 494,455 | 783,858 | $0.21 |
| spp | 1,803 | 697,609 | 166,658 | 864,267 | $0.10 | |
| DSPy | 2,084 | 1,283,741 | 318,806 | 1,602,547 | $0.19 | |
| TREC | EvoPrompt | 4,688 | 246,696 | 575,620 | 822,316 | $0.24 |
| spp | 2,303 | 1,783,177 | 179,820 | 1,962,997 | $0.11 | |
| DSPy | 1,577 | 748,696 | 345,665 | 1,094,361 | $0.18 |
Three distinct profiles:
- spp is input-heavy / output-light. One rich, static, six-section prompt replayed per row; the model emits a single label word, so output tokens stay tiny (43k–180k). On TREC its total token count (1.96M) is the highest of any cell in the table — yet it is the cheapest arm on that task ($0.11), because nearly all those tokens are cheap input, and the ~1.3k-token prompt crossed gpt-5-nano’s 1,024-token caching threshold (cached input bills ~10× cheaper). Counting tokens would call spp the loser here; counting dollars shows it the winner.
- EvoPrompt is output-heavy. GA generations of new prompt text plus per-call reasoning push output to 400k–576k — and output is the 8×-expensive token — so its modest total token counts still cost the most on two of three tasks.
- DSPy is input-heavy via demos. Bootstrapping few-shot demonstrations balloons input (0.75M–1.33M) and runs the highest total token counts overall; the demo tokens are cheap input, so its dollar cost lands in the middle.
Takeaway: report dollars, not tokens. Any “total tokens” leaderboard would mis-rank every task.
4. Fairness ledger — what cuts against spp
A benchmark that only flatters one arm is not credible. Here is the other side.
- spp’s optimizer cost is not billed in these dollars. This ledger counts gpt-5-nano task-model tokens only. EvoPrompt and DSPy spend their entire optimization budget on that task model, so their full cost is visible here. spp offloads the optimization reasoning to Claude subagents and a human in the loop — that labor is real and is not in the $0.25. The honest claim is therefore narrow: on task-model spend, spp is 2.5× cheaper. It is not a claim that spp is cheaper end-to-end once human/Claude time is priced in. For a fully-automated, walk-away pipeline, DSPy and EvoPrompt have no hidden second budget; spp does.
- DSPy ran few-shot; the other two ran 0-shot. DSPy’s accuracy edge on AG News (and its parity on SST-5) comes partly from in-context demos the other arms never used. Conversely, that spp matched DSPy 0-shot on SST-5 is the genuinely interesting finding — but credit DSPy that its regime is doing real work.
- TREC’s spp cost benefited from prompt caching the other two could not get (their prompts are below the 1,024-token threshold). spp’s win on TREC cost is partly a structural side-effect of its longer prompt, not pure efficiency. The list-price (un-cached) estimate for spp-TREC would be ~$0.16, still below DSPy’s $0.18 and EvoPrompt’s $0.24 — so the ranking holds, but the margin is caching-aided.
- spp’s TREC ledger honestly includes ~50k “calibration” tokens (~2.6%) spent before its harness was aligned to EvoPrompt’s exact format. Counted, not hidden.
- spp’s AG News R&D exploration (run_05, +240 calls) is excluded from its shipped total — correctly, since it was rejected and never shipped, but worth stating that the $0.04 is the shipped-arm figure, not every token ever spent on the task.
- TREC used 500 test rows, not 1,000. Its standard error is larger (~0.014), so spp’s 5-point TREC margin over DSPy, while still ~3.5 SE and significant, rests on half the sample of the other two tasks.
None of these reverse the headline. They bound it: spp matches or beats both automated arms on accuracy and is materially cheaper on task-model spend — provided you account separately for the human/Claude effort it shifts off the task model.
5. What each arm is actually good at
- spp — best mean accuracy, cheapest task-model spend, and the only arm to post a decisive accuracy win (TREC, +5 over the nearest competitor). Its strength is writing one sharp categorical rule that a saturated seed was missing (science→Tech on AG News; the answer-type decision order on TREC) and then stopping. Its hidden cost is the human/Claude loop.
- DSPy (MIPROv2, few-shot) — the strongest fully-automated arm: highest accuracy on AG News, tied-best on SST-5, no human in the loop. Its lever is bootstrapped demonstrations. Costs ~2× spp’s task-model spend and runs the most total tokens.
- EvoPrompt (GA, 0-shot) — did not beat its own human seed on any task on this model, and regressed on TREC. Output-heavy and the most expensive arm on two of three tasks. On a strong instruction-following model like gpt-5-nano, GA prompt mutation had little headroom to exploit.
6. Design signal for spp (the “incorporate the gap” question)
The one place an automated arm clearly out-designed spp was AG News, by 0.5 pt, via few-shot demos — inside the noise band, but it points at a real lever spp does not use: in-context demonstrations. On SST-5, spp’s 0-shot loop already matched DSPy’s few-shot, suggesting spp’s rule-writing substitutes for demos rather than needing them. The honest design takeaway: an optional, auditor-gated few-shot mechanism is a legitimate v2.0 spp candidate — demos that encode a rule (one canonical exemplar per hard class), not row-fitting — to be tested against the hypothesis that spp’s 0-shot rules already capture most of what demos provide.
All figures sourced from results/<arm>/<task>/result.json, baselines/<task>/spp/<task>/runs/gpt-5-nano/finalize/test_eval.json, and the per-task baselines/<task>/token_usage.md ledgers (dashboard-verified). See Reproduce to regenerate the tables.