Stats — definitions, benchmarks, how to compute

Stats prove the loop works and tell where to spend the next fix. Report them after every round.

Definitions

Round classification (when no rounds.json): verdict text "first try" → 1; "redo 1", "(redo)", "both tries", "= sell" → 2; "after fix", "try 1 validated" → 3; anything else → 4 + warning.

Compute

SAC_WORKDIR=/root/workspace/<brand> python3 /root/claude-vault/skills/static-ads/scripts/stats.py --ads 83

Writes $SAC_BATCH/stats.json. build_sam.py shows a baseline column if $SAC_WORKDIR/stats_baseline.json (or SAC_BASELINE) exists — copy a previous batch's stats.json there to compare runs.

Run-1-style stats (no calls log, several batch folders): the Wildhorn stats.py counted every image file per ad (out/<key>_v*.png + _sd_ + _raw_) plus "moderation_blocked" lines in logs, and rounds = number of batches in which the ad got a version. Use calls.jsonl from day one instead.

Benchmarks (Wildhorn)

Per round in the Sam batch: round 1 6/10 first try (60 %), round 2 26/34 new ads (76 %), round 3 31/43 new ads (72 %); QC round 3 fixed 13 defects before review.

What moved the numbers (in order of impact, from the case studies): the pixel inventory (A13) → 0 % to 60–76 % first try; "change the minimum + one-word angle swap" (E7–E10); scene-only Seedream + composite for refusals (D7) → no garbled sexual ads; frozen tries + sell overrides (A20/A26) → no regression on redos; the blocking lint (D-rules).

Known measurement caveats

• calls.jsonl undercounts the Sam batch slightly: the A25 steps for S11 (extra Seedream edits s_11_sd_v4…v6 and the jarswap.py call) were run by hand and not logged (5 calls logged for S11; ~3–4 more were made). Log manual calls by hand.
• Two "first try" figures exist on the Wildhorn pages: /sam showed 67.5 % (counted ads with exactly one output file = 56/83), /sam-final showed 71.1 % (verdict text = 59/83). The difference is S38, S43 and S71: each got a second file from my own QC fix BEFORE the reviewer saw it (avatar age, misspelled logo, leftover template prop), then was validated on its first review. Use the verdict-based number (it measures the reviewer's first look); report pre-review QC fixes separately (13 in Sam round 3).
• Run 1 "first try" is 0 %: every one of the 79 ads got at least one later version after the reviewer's batch-01 feedback (rounds = number of batches with a version; minimum 2).
• The /sam page's run-1 rounds line showed "2 rounds: 0 ads" because stats_run1.json stores the distribution with string keys and the page looked them up with int keys (cosmetic bug in the Wildhorn builder; fixed in the copied builders by not reading that file).

Targets for a new brand

• First batch of 10: expect 40–60 % first try (the rules are generic; brand facts and taste are new). By round 3: ≥ 70 %.
• Hit rate ≥ 55 %, generations per validated ad ≤ 2.
• Any metric that drops vs the previous round → look at which rule the misses share, not at the engine.

Run 1 round history (from the checkpoint log, 01/10/2026)

Every redo went back on the review page with all its earlier versions; 79/79 were final after Loom 9 (run1-final/, #74 = its B8 version).

What the Sam batch ledger says (where first tries are lost)

Reading: the refusal path is where first tries die (it was also where D6, D7 and A25 were learned). For a new brand, plan every sexual/skin template at spec time with a complete scene recipe (sd_prompt with empty zones, composite with every placement), or choose the A25 method directly. Sources with mostly-UGC/photo templates with people (Sugar-Splash 1/4, Resilia 1/3) were harder than designed posters (Everly 5/5, Spicy-Cubes 9/10).

Sam batch ledger (all 83 ads)

Generated from sam-batch/s*specs.py (last spec per ad), templates/ (rank = ad number), calls.jsonl, validated.json, sell_override.json. "Tries" = output files out/s_NN_vK.png; "Calls" = engine calls logged (a refused call produces no file; QC fixes produce files).