Model Performance Report
Critical analysis across 6 dispatch phases, 3 models, 27 gates. Built from run traces, MANIFEST.json files, GATE_ANALYSIS.md reports, dispatch specs, and output quality spot-checks.
Executive Summary
Across 6 phases, 3 models, and 27 gates, the loops-explorer-app build achieved a 92.6% gate pass rate (25/27) with A-quality output in 5 of 6 phases. deepseek-v4-pro emerged as the most reliable model: 103 tool calls across 2 phases, 6,864 output files, zero reformulations, and 23-34% budget utilization — the definition of efficiency. mimo-v2.5-pro demonstrated the highest design quality (pure SVG charts, clean Python) but stumbled on Phase 6 when given only 50 tool calls for a Cloudflare deployment. minimax-m3 produced publication-quality analysis (12 patterns, 9 anti-patterns, 20 prompt scores) but consumed 79% of its budget — the least efficient tool-call-to-output ratio. The single critical failure — Phase 6 deploy HALTED with 404 — was caused by budget starvation (50 calls vs the 120-200 standard) and a thinkingLevel mismatch (low on a debugging-heavy task). This is an orchestrator-level error, not a model capability failure. The fix is two-fold: (1) allocate deployment phases ≥120 calls, and (2) checkpoint the CLAUDE_DIGEST before attempting deployment. The build validates a core hypothesis: model-task matching plus adequate budgets plus concrete gates produces reliable, high-quality output. The one exception proves the rule: when any of those three pillars breaks (budget, in this case), the output degrades. The pipeline is sound; the budget allocation for Phase 6 was not.
Per-Phase Performance
| Ph | Name | Model | Calls | Budget | % Used | Time | Att | Gates | Quality |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Ingest Pipeline | mimo-v2.5-pro | 23 | 120 | 19.2% | 177s | 1 | 6/6 | A |
| 2 | Astro Scaffold | deepseek-v4-pro | 35 | 150 | 23.3% | 134s | 1 | 8/8 | A |
| 3 | Route Pages | deepseek-v4-pro | 68 | 200 | 34.0% | 406s | 1 | 6/6 | A |
| 4 | Analytics & Search | mimo-v2.5-pro | 31 | 150 | 20.7% | 191s | 1 | 5/5 | A |
| 5 | Prompt Analysis | minimax-m3 | 79 | 100 | 79.0% | 430s | 1 | 4/4 | B+ |
| 6 | Cloudflare Deploy | mimo-v2.5-pro | 50 | 50 | 100.0% | 357s | 2 | 1/3 | D |
Phase 1Ingest Pipelinemimo-v2.5-proA
Exemplary ingest script. Used only 19% of budget. Well-documented design decisions. Strong use of SKIP_DIRS/SKIP_FILES/SKIP_EXTENSIONS constants.
Output: 5,814 entries, 26 categories, 10 subprojects. Clean Python with constants, skip patterns, extension/category maps. 1,114,829 lines scanned.
Gates
- ingest.py exists
- manifest.json has ≥500 entries
- stats.json has totals with bySubproject
- entries dir count matches manifest
- categories.json has ≥5 categories
- CLAUDE_DIGEST exists
Phase 2Astro Scaffolddeepseek-v4-proA
Fastest phase at 134s. Clean types.ts with proper union types and interfaces. Most gates of any phase (8). Build 0 errors.
Output: 12 files: config (6), types (1), data layer (2), UI (3). Astro 7.1.3, OKLCH dark theme, 7 nav links, Pagefind indexed.
Gates
- package.json with astro
- astro.config.mjs exists
- types.ts has FileEntry
- data.ts imports manifest
- entries.ts has getEntry
- BaseLayout with OKLCH
- wrangler.jsonc exists
- astro build exits 0
Phase 3Route Pagesdeepseek-v4-proA
Most complex phase — generated 6,748 SSG pages. Fixed browser compat (color-mix), filtered broken tags with bash interpolation, deterministic SubprojectBadge colors. Used only 34% of budget despite complexity.
Output: 6 components, 12 pages, 9 route families, 6,748 total pages. Pagination, tag cloud, prev/next nav, ContentPreview.
Gates
- FileCard component exists
- FileGrid component exists
- file detail page with getStaticPaths
- subproject routes exist
- language routes exist
- build exits 0 with ≥100 pages
Phase 4Analytics & Searchmimo-v2.5-proA
Architectural taste: all SVG charts server-rendered with aria-labels and desc elements. OKLCH color tokens. FileTypeDonut with proper arc math. Gate found 6 charts (≥4 required).
Output: 5 chart components (all pure SVG, zero client JS), analytics page with stat tiles + 2×2 grid, search page with Pagefind UI. 6 charts detected by gate.
Gates
- analytics page exists
- search page exists
- 4+ chart components
- pagefind attributes on detail page
- build + pagefind runs
Phase 5Prompt Analysisminimax-m3B+
Comprehensive analysis but used 79% of budget. Gate analysis flagged SIMPLIFY: inline Python one-liner over 300 chars (belongs in script file). Only model assigned thinkingLevel: high. Most budget-intensive model per call (0.79 calls/budget ratio).
Output: 12 patterns, 9 anti-patterns, 20 prompt scores, 10 recommendations. 5-dimension scoring. 25.6 KB JSON. Sortable table page.
Gates
- prompt-analysis.json exists with patterns
- prompt scores cover ≥15 prompts
- prompt-analysis.astro page exists
- build exits 0
Phase 6Cloudflare Deploymimo-v2.5-proD
CRITICAL: Severely under-budgeted at 50 calls vs 120-200 for other phases. Budget exhausted on attempt 2. Deployment likely incomplete — HTML built but Cloudflare Workers deployment or routing not configured. Model was also set to thinkingLevel: low, which may have impaired its ability to debug the 404.
Output: HALTED (BUDGET_TOOLCALLS). dist/ has 6,751 HTML files but live URL returned 404. CLAUDE_DIGEST not written. 2 of 3 gates failed.
Gates
- dist has HTML files
- live URL returns 200
- CLAUDE_DIGEST exists
Cross-Model Comparison
Most Efficient
deepseek-v4-pro103 total tool calls across 2 phases, producing 6,864 output files (12 scaffold + 6,852 pages). That's ~0.015 calls per output file. Both phases used <35% of budget. Fastest individual phase at 134s (Phase 2).
Highest Quality
mimo-v2.5-promimo-v2.5-pro produced the clearest evidence of architectural taste: Phase 4's pure SVG charts with zero client JS, aria-labels, and CSS-variable-driven color tokens. Phase 1's ingest.py with clean constants, path normalization, and SKIP_DIRS/SKIP_FILES/SKIP_EXTENSIONS shows production-grade Python discipline. However, Phase 6's budget-exhaustion failure drags its reliability score down.
Best Compliance
deepseek-v4-prodeepseek-v4-pro used only 23-34% of budget across both phases, never needed reformulations, and passed all 14 gates. The Phase 3 build produced 6,748 pages with zero errors and included proactive fixes for browser compat (color-mix) and bad data (bash interpolation in tags). This model treats the spec as a contract.
Best Creative Analysis
minimax-m3minimax-m3's Phase 5 prompt-analysis report was the most analytical task in the build. It scored 20 prompts across 5 dimensions, identified 12 patterns and 9 anti-patterns, and produced 10 actionable recommendations. The output is publication-quality. However, it used 79% of budget (least efficient) and tripped one SIMPLIFY gate-format warning.
Per-Model Aggregates
| Model | Phases | Total Calls | Total Budget | % Used | Total Time | Gates | Status |
|---|---|---|---|---|---|---|---|
| mimo-v2.5-pro | 3 | 104 | 320 | 32.5% | 725s | 12/14 | ⚠️ Has failure |
| deepseek-v4-pro | 2 | 103 | 350 | 29.4% | 540s | 14/14 | ✅ All good |
| minimax-m3 | 1 | 79 | 100 | 79.0% | 430s | 4/4 | ✅ All good |
Dispatch Recommendation
deepseek-v4-pro is the workhorse — assign it scaffolding, routing, and schema tasks. mimo-v2.5-pro is the craftsman — assign it visualization, data pipelines, and design-heavy phases (with adequate budget). minimax-m3 is the analyst — assign it scoring, pattern detection, and report-generation tasks (with budget expectations set appropriately). Critical rule: never under-budget a deployment phase.
Effective Patterns 7
Budget-Matched Model Selection
highEach model was assigned to phases matching its established strengths: deepseek-v4-pro for structural/architectural tasks (scaffold, routing), mimo-v2.5-pro for visual/generative tasks (charts, ingest), minimax-m3 for analytical/creative tasks (prompt analysis). This discipline produced A-quality in 5/6 phases.
Phase 2-3 (deepseek): clean types.ts, 6748 SSG pages, 34% budget use. Phase 1,4 (mimo): pure SVG charts, well-structured ingest. Phase 5 (minimax): comprehensive prompt scoring across 5 dimensions.
SVG-First Zero-JS Architecture
highPhase 4 (mimo-v2.5-pro) produced 5 chart components using pure SVG with server-side rendering. No client JavaScript for charts. Donut chart uses proper arc math with aria-labels and desc elements. Color tokens pulled from CSS variables. This eliminates runtime dependencies and ensures charts render even with JS disabled.
FileTypeDonut.astro: arcPath() function with canvas-like math, <title> and <desc> for accessibility, OKLCH variable-based color tokens. All 6 chart components follow same pattern.
Validation Gate Density Correlates with Quality
highPhases with the most gates also had the highest output quality. Phase 2 had 8 gates (most of any phase) and produced a flawless scaffold. Phase 3 had 6 gates including the critical build-exits-0-with-≥100-pages check that verified 6,748 output pages. Gates serve as both validation and specification — they tell the model what 'done' means.
Phase 2: 8 gates, 0 failures, A-quality. Phase 3: 6 gates, 0 failures, A-quality. Phase 6: 3 gates, 2 failures, D-quality. Gates are load-bearing specification elements.
Budget Headroom = Quality Multiplier
highPhases using <25% of their budget produced A-quality output. Phase 6 at 100% budget produced D-quality. The sweet spot appears to be 20-35% budget utilization — enough to do the work plus handle unexpected edge cases without panic.
Phase 1: 19% used → A. Phase 2: 23% → A. Phase 4: 21% → A. Phase 3: 34% → A. Phase 5: 79% → B+. Phase 6: 100% → D (HALTED).
Gate Analysis as Preflight Guard
mediumThe GATE_ANALYSIS.md step in each dispatch caught issues before execution. Phase 5's SIMPLIFY flag (inline Python one-liner over 300 chars in gate command) was caught during preflight. While it didn't block the phase, it surfaced a maintainability anti-pattern that can be fixed in a future pass.
Phase 5 GATE_ANALYSIS.md: 'Inline python one-liner over 300 chars — belongs in a script file for maintainability and testability.' Phases 1-4,6 showed no issues — confirming gates were well-formed.
Single-Attempt Completion Rate
high5 of 6 phases completed on the first attempt with zero reformulations. This indicates the dispatch spec → model alignment is strong. Only Phase 6 (deploy) required a reformulation attempt and still failed, suggesting the failure was structural (budget under-provisioning) rather than model capability.
Phases 1-5: attempts=1, reformulations=0. Phase 6: attempts=2, reformulations=1, still HALTED.
CLAUDE_DIGEST Output Format Discipline
highAll successful phases produced a CLAUDE_DIGEST.md with Status, Attempts, Tool calls, Wallclock, Validation table, and explicit 'Phase N Complete' summary with deliverables table. This consistent format enables the orchestrator's cross-phase analysis. Phase 6 failed to produce one, which is itself a diagnostic signal.
Phases 1-5: all have CLAUDE_DIGEST.md with uniform structure. Phase 6: CLAUDE_DIGEST exists gate failed with exit code 1 — probably never written.
Anti-Patterns 7
Budget Starvation on Complex Tasks
criticalPhase 6 (Cloudflare Deploy) was allocated only 50 tool calls — the lowest budget of any phase — for a task that involves: building the site, deploying to Cloudflare Workers via wrangler, verifying the live URL, and writing a digest. This is inherently multi-step and requires Wrangler API interactions. The model exhausted its budget on the second attempt without reaching the live-URL or digest gates.
Phase 6 spec shows budget.maxToolCalls=50 vs Phase 1 (120), Phase 3 (200), Phase 5 (100). Despite being the most operations-heavy phase (build + deploy + verify), it got the tightest budget. Result: HALTED with 2/3 gates failing.
Thinking Level Mismatch on Debug-Heavy Tasks
highPhase 6 was assigned thinkingLevel: 'low' despite being a deployment task that predictably requires debugging (DNS, Wrangler config, Cloudflare routing, environment variables). The 404 on the live URL suggests a misconfiguration that deeper reasoning might have caught. Compare: Phase 5 (analysis) got thinkingLevel: 'high' when the task was primarily reading and scoring — not debugging infrastructure.
Phase 6 spec: thinkingLevel=low. Phase 5 spec: thinkingLevel=high. Deployment failures (DNS, routing, Wrangler config) are the most common source of 404s on Cloudflare Workers. Low thinking level may have prevented the model from systematically debugging each layer.
Budget Exhaustion Without Early Artifact Save
highWhen Phase 6 hit BUDGET_TOOLCALLS, it had built dist/ with 6,751 HTML files but never wrote CLAUDE_DIGEST.md. This suggests the model didn't save partial progress before exhausting budget. A disciplined model would checkpoint: write the digest after the build succeeds, even if deployment fails.
dist has HTML files gate passed (6751 files), but CLAUDE_DIGEST gate failed with exit code 1 (file doesn't exist). The build succeeded — a CLAUDE_DIGEST should have been written before attempting deployment.
Gate Analysis Ran But Didn't Prevent Failure
mediumPhase 6's GATE_ANALYSIS.md reported 'No issues found. All gates are clean.' Yet 2 of 3 gates failed at runtime. The gate analysis checked gate structure (well-formed commands) but didn't evaluate gate feasibility given the budget constraints. A smarter preflight would have flagged: '50 tool calls for build+deploy+verify is tight — consider splitting into build phase and deploy phase.'
Phase 6 GATE_ANALYSIS.md: 'No issues found. All gates are clean.' Runtime: live URL 404, CLAUDE_DIGEST missing. The analysis was structurally correct but contextually blind.
Inline Python in Gate Commands
lowPhase 5's prompt-analysis gate used an inline Python one-liner exceeding 300 characters for validation. The GATE_ANALYSIS flagged this as SIMPLIFY. While the gate passed, long inline commands are brittle, hard to debug, and violate the 'gates should be simple bash' principle.
Phase 5 GATE_ANALYSIS.md: 'Inline python one-liner over 300 chars — belongs in a script file for maintainability and testability.' The command: python3 -c "import json; d=json.load(open(...)); assert len(d.get('patterns',[])) >= 5..."
minimax-m3 Budget Intensity
mediumminimax-m3 consumed 79% of its budget (79/100 calls) on a single phase — the highest budget utilization of any model-phase combination. For comparison, deepseek-v4-pro's heaviest phase used 34%. While the output was comprehensive (12 patterns, 9 anti-patterns, 20 scores), the budget burn rate suggests minimax-m3 may be less efficient at tool-call triage, potentially over-reading files when a scan would suffice.
minimax-m3 Phase 5: 79/100 calls (79%). deepseek-v4-pro Phase 3: 68/200 calls (34%) — nearly identical call count but half the budget utilization. Could indicate less disciplined read-first-triaging.
Deploy Gate Ordering Gap
mediumPhase 6 gates check build → live URL → CLAUDE_DIGEST in that order. However, CLAUDE_DIGEST is the orchestrator's primary consumption artifact and should be verified first or second. The current ordering means: if deployment fails (live URL 404), the digest never gets written, and the orchestrator loses the failure narrative. The gate should be: build → CLAUDE_DIGEST → deploy → live URL.
Phase 6 MANIFEST shows validation labels: 'dist has HTML files' → 'live URL returns 200' → 'CLAUDE_DIGEST exists'. If the digest were checked before the live URL, the orchestrator would at least have the failure narrative even if deployment failed.
Gate Performance Detail
| Ph | Gate | Model | Result | Useful? |
|---|---|---|---|---|
| 1 | ingest.py exists | mimo-v2.5-pro | ✅ Pass | Sanity |
| 1 | manifest.json has ≥500 entries | mimo-v2.5-pro | ✅ Pass | Useful |
| 1 | stats.json has totals with bySubproject | mimo-v2.5-pro | ✅ Pass | Useful |
| 1 | entries dir count matches manifest | mimo-v2.5-pro | ✅ Pass | Useful |
| 1 | categories.json has ≥5 categories | mimo-v2.5-pro | ✅ Pass | Useful |
| 1 | CLAUDE_DIGEST exists | mimo-v2.5-pro | ✅ Pass | Sanity |
| 2 | package.json with astro | deepseek-v4-pro | ✅ Pass | Useful |
| 2 | astro.config.mjs exists | deepseek-v4-pro | ✅ Pass | Sanity |
| 2 | types.ts has FileEntry | deepseek-v4-pro | ✅ Pass | Useful |
| 2 | data.ts imports manifest | deepseek-v4-pro | ✅ Pass | Useful |
| 2 | entries.ts has getEntry | deepseek-v4-pro | ✅ Pass | Useful |
| 2 | BaseLayout with OKLCH | deepseek-v4-pro | ✅ Pass | Useful |
| 2 | wrangler.jsonc exists | deepseek-v4-pro | ✅ Pass | Sanity |
| 2 | astro build exits 0 | deepseek-v4-pro | ✅ Pass | Critical |
| 3 | FileCard component exists | deepseek-v4-pro | ✅ Pass | Sanity |
| 3 | FileGrid component exists | deepseek-v4-pro | ✅ Pass | Sanity |
| 3 | file detail page with getStaticPaths | deepseek-v4-pro | ✅ Pass | Useful |
| 3 | subproject routes exist | deepseek-v4-pro | ✅ Pass | Useful |
| 3 | language routes exist | deepseek-v4-pro | ✅ Pass | Useful |
| 3 | build exits 0 with ≥100 pages | deepseek-v4-pro | ✅ Pass | Critical |
| 4 | analytics page exists | mimo-v2.5-pro | ✅ Pass | Sanity |
| 4 | search page exists | mimo-v2.5-pro | ✅ Pass | Sanity |
| 4 | 4+ chart components | mimo-v2.5-pro | ✅ Pass | Useful |
| 4 | pagefind attributes on detail page | mimo-v2.5-pro | ✅ Pass | Useful |
| 4 | build + pagefind runs | mimo-v2.5-pro | ✅ Pass | Critical |
| 5 | prompt-analysis.json exists with patterns | minimax-m3 | ✅ Pass | Sanity |
| 5 | prompt scores cover ≥15 prompts | minimax-m3 | ✅ Pass | Useful |
| 5 | prompt-analysis.astro page exists | minimax-m3 | ✅ Pass | Sanity |
| 5 | build exits 0 | minimax-m3 | ✅ Pass | Critical |
| 6 | dist has HTML files | mimo-v2.5-pro | ✅ Pass | Useful |
| 6 | live URL returns 200 | mimo-v2.5-pro | ❌ Fail | Blocked |
| 6 | CLAUDE_DIGEST exists | mimo-v2.5-pro | ❌ Fail | Sanity |
27 gates total — 25 passed (92.6%), 2 failed. The build-exits-0 gates in Phases 2, 3, 4, 5 were the most load-bearing: they caught real issues (color-mix CSS, bad tags, Pagefind indexing). File-existence gates served as sanity checks. The live-URL-200 gate in Phase 6 was well-designed but couldn't be reached due to budget exhaustion.
Recommendations
- Never under-budget a deployment phase. Cloudflare Workers deployment involves Wrangler API calls, DNS verification, and routing configuration — allocate ≥120 tool calls for any phase touching Cloudflare infrastructure.
- Match thinkingLevel to task type, not perceived difficulty. Deployment tasks need high thinkingLevel for debugging; analytical tasks (reading + scoring) work fine at medium. Phase 6's low thinkingLevel on deploy was the wrong call.
- Checkpoint CLAUDE_DIGEST.md after the build step, before attempting deployment. The orchestrator needs the failure narrative even when deployment fails.
- Reorder Phase 6 gates: build → CLAUDE_DIGEST → deploy → live URL. The digest is the orchestrator's consumption artifact and should gate on success before attempting deployment.
- deepseek-v4-pro is the clear winner for structural tasks (scaffold, routing, types) — assign it all Phase 3-equivalent work. It delivers A-quality at 23-34% budget utilization.
- Set minimax-m3's budget to 120+ for analytical tasks. Its 79% burn rate on Phase 5 shows it reads more, which produces comprehensive output but needs headroom.
- Continue the SVG-first zero-JS pattern from Phase 4. It's the strongest architectural decision in the app — accessible, performant, and framework-independent.
- Add a preflight budget feasibility check to the orchestrator: if gate count × 15 > budget maxToolCalls, flag it. A 3-gate deploy phase needs at minimum ~45 calls just for gate execution, leaving only 5 for actual work.