Building an Eval Harness That Catches Regressions
Anthropic shipped three concurrent regressions over six weeks and their eval suite caught none of them. Even Anthropic ships blind. Here is the six-layer harness pattern that would have caught it.
blog
Deep dives and field notes on local-first AI, agentic architecture, and what is actually working in 2026, with primary sources and reproducible benchmarks.
Showing 15 of 38 posts in Production · clear
Long-form research articles with primary sources, benchmarks, and reference tables.
Anthropic shipped three concurrent regressions over six weeks and their eval suite caught none of them. Even Anthropic ships blind. Here is the six-layer harness pattern that would have caught it.
Of the ~14,000 MCP servers in PulseMCP's hand-curated index, fewer than 30 are demonstrably production-ready. Here is the list, the criteria, and the failure modes.
Nine frameworks, one durable-execution wedge, zero unbroken benchmarks. An honest map of the multi-agent ecosystem in May 2026, anchored in Anthropic's 90.2%/15× receipts and Berkeley's eight-benchmark exploit.
$9/mo of Hetzner replaces $80 to $150/mo of SaaS. Listmonk, Cal.com, Umami, and Uptime Kuma on a single CX22, with the deliverability playbook nobody writes down.
The 10 things from AI Agent Conference 2026 NYC (May 4-5, NY Hilton Midtown) that are actually load-bearing if you ship agents in 2026. The trust paradox, CrewAI's 42% AI-authored code, the iceberg under every project, AX as the new UX, and what the panels from Datadog, LanceDB, Carta, and the Codex/Linear/Graphite room actually said.
The 10 things from AI Dev 26 SF (April 28-29, Pier 48) that are actually load-bearing if you build agentic systems in 2026. Marc Brooker on defects, Andrew Ng on PM bottlenecks, Bain's 8-subgraph payroll system, the 4-legged identity, hybrid doc OCR, and the simulation sandbox every action-taking agent needs.
Codex, Linear, and Graphite shared the stage at AI Agent Conference NYC on what scales coding agents past the demo. The infrastructure underneath is the actual work.
The single highest-leverage decision when shipping mission-critical autonomous agents. Production is the only truth.
Why five specialized $0.01 agents beat one $0.50 god model, and what the multi-agent crowd gets wrong about it.
Why human-in-the-loop is the only ethical and profitable way to scale agentic AI in a world of bot fatigue.
The hidden engineering that decides whether your agent makes it to production. The 65/95 gap and the three foundations underneath it.
90% of enterprise data is locked in PDFs. The 2026 pipeline that gets it out is not RAG, not vision-only, and not the OCR you remember from 2018.
Why bigger context windows are not the answer, and what production-tuned engineers actually trust in 2026.
The fastest 2026 teams are testing autonomous agents in synthetic enterprise environments before any customer is exposed. With the case for it and the open-source pieces to build one.
Agentic loops that detect, diagnose, and fix deployment errors before you see the notification. With the workflow that actually works in 2026.