Loop log — AG News
Topic classification (World / Sports / Business / Tech). 4 iterations, one rule that mattered.
Task model: gpt-5-nano (reasoning_effort low), locked. Internal splits: train 80 / dev 80, balanced. Sacred test: 1,000 rows, read once at finalize. Full report: baselines/ag_news/spp/ag_news/runs/gpt-5-nano/REPORT.md.
The shape of this run
AG News is near-saturated — the bare seed already scored ~0.87, and EvoPrompt’s genetic search found nothing better than the seed itself. So the loop’s job was narrow: find the one categorical rule the seed was missing, add it, and stop before fitting noise. It did exactly that in four iterations.
The interaction, gate by gate
The run as a human-agent exchange — the user’s approval phrases (verbatim from plan.md) and what the agent did in response.
| Step | What the user said | What the agent did |
|---|---|---|
| G1 | “approved, proceed to baseline” | canonicalized the baseline and audited the imported gold → verdict ready, exactly class-balanced (250 / 250 / 250 / 250) |
| G2 | “approved, proceed to splits” | generated stratified splits: train 80 / dev 80 (20 per class), verified 0 baseline↔︎test overlap |
| G3 | “approved, start the loop” | built prompt_v01 (bare seed + output directive only); dry-run on 3 rows passed 3/3 |
| G4 | “approved, run iteration 1” | ran 4 iterations (see trajectory below); EARLY_STOP, selected prompt_v03 |
| — | “explore further” (post-loop R&D) | tested 4 few-shot demos → dev 0.8625 (regressed); chain-of-thought at reasoning=medium → dev 0.925 but ~5× output cost, breaks the 0-shot apples-to-apples — both rejected, v03 unchanged |
| G5 | “approved, score the test set” | one-time sacred read (1,000 rows) → test 0.876, 0 parse failures |
| G6 | “approved, freeze the prompt” | froze prompt_v03 → PROMPT_FROZEN_v01.md (SHA-256 9b7a1fd9…baf96, byte-verified) |
Dev trajectory
| Iter | Prompt | Edit(s) | Dev | Train | Decision |
|---|---|---|---|---|---|
| 1 | v01 | bare seed + output-format directive | 0.8875 | 0.8750 | baseline |
| 2 | v02 | +Edit 1 (science→Tech) +Edit 2 (company-subject) | 0.8750 | 0.8750 | Edit 2 caused collateral → revert |
| 3 | v03 | Edit 1 only (drop Edit 2) | 0.9125 | 0.8875 | best — selected |
| 4 | v04 | +Edit 4 (narrow Tech-pull company rule) | 0.9000 | 0.8875 | regressed → rejected |
What happened, iteration by iteration
Iteration 1 — establish the baseline. The prompt was the identical EvoPrompt seed (“categorize into World, Sports, Business, or Tech”) plus a bare output-format directive so the exact-match scorer could parse the label. No definitions, no examples. Dev 0.8875.
Iteration 2 — the first edit, and a lesson in collateral. The discrepancy stage found science/nature stories (“global warming”, “rare whale”, “dinosaur fossil”) being routed to World instead of Tech. It proposed Edit 1: Tech = science and technology. It also bundled Edit 2, a “tech-company-subject” rule with a “purely financial → Business” clause. v02 dev dropped to 0.8750 — the financial clause mis-fired (“Renault F1 launches…” → Business). The diagnosis: the collateral came from the Business-pushing half of Edit 2, not the Tech-pulling half.
Iteration 3 — revert the bad half, keep the good rule. (Selected.) The loop dropped Edit 2 entirely and kept only the science clarification. The score-blind auditor was asked the hard question — is reverting a rule just score-chasing? — and ruled that removing an over-generalizing rule is itself categorical and leaves a simpler prompt, the opposite of overfitting. Dev rose to 0.9125, the peak.
Iteration 4 — try the last boundary, fail honestly. The only remaining signal was a Tech→Business cluster (computing companies whose news peg is a corporate event: “IBM To Spin Off PC Unit”). Edit 4 added a Tech-pulling-only company rule. Dev regressed to 0.9000 — because the gold here is internally contradictory (IBM/Apple corporate news is labeled Tech, but telecom/Cingular corporate news is labeled Business; no rule satisfies both).
Why it stopped
A single clear dev peak at v03, with the only remaining boundary giving contradictory gold. Continuing would mean iterating on ±1-row noise over an 80-row dev — overfitting. The loop early-stopped and selected v03. Recognizing label noise and stopping is the method working.
The measured contribution
One clean, one-directional categorical rule the seed lacked: Tech = science AND technology (research, nature, space, medicine, the internet — not gadgets only). Everything else was the seed’s existing behavior.
Result and generalization
- Test accuracy 0.876 on 1,000 sacred rows (0 parse failures).
- Dev 0.9125 → test 0.876, a −0.036 gap within the 80-row dev sampling noise (SE ≈ 0.032).
- The one residual soft spot (Tech→Business) is exactly the contradictory-gold boundary the loop correctly declined to over-fit.
Integrity: every edit verdicted categorical, 0 overrides, overfit guard never tripped (dev ran above train throughout), sacred test untouched until finalize. See the Results page for how this lands against EvoPrompt and DSPy.