Building an Eval Harness That Catches Regressions
Anthropic shipped three concurrent regressions over six weeks and their eval suite caught none of them. Even Anthropic ships blind. Here is the six-layer harness pattern that would have caught it.
blog
Deep dives and field notes on local-first AI, agentic architecture, and what is actually working in 2026, with primary sources and reproducible benchmarks.
Showing 29 of 38 posts in Agent architecture · clear
Long-form research articles with primary sources, benchmarks, and reference tables.
Anthropic shipped three concurrent regressions over six weeks and their eval suite caught none of them. Even Anthropic ships blind. Here is the six-layer harness pattern that would have caught it.
Of the ~14,000 MCP servers in PulseMCP's hand-curated index, fewer than 30 are demonstrably production-ready. Here is the list, the criteria, and the failure modes.
An 84% cost reduction on a real SaaS workload, a 97% reduction on agentic dev loops, and the three-tier mix that actually ships in May 2026.
Nine frameworks, one durable-execution wedge, zero unbroken benchmarks. An honest map of the multi-agent ecosystem in May 2026, anchored in Anthropic's 90.2%/15× receipts and Berkeley's eight-benchmark exploit.
The 10 things from AI Agent Conference 2026 NYC (May 4-5, NY Hilton Midtown) that are actually load-bearing if you ship agents in 2026. The trust paradox, CrewAI's 42% AI-authored code, the iceberg under every project, AX as the new UX, and what the panels from Datadog, LanceDB, Carta, and the Codex/Linear/Graphite room actually said.
The 10 things from AI Dev 26 SF (April 28-29, Pier 48) that are actually load-bearing if you build agentic systems in 2026. Marc Brooker on defects, Andrew Ng on PM bottlenecks, Bain's 8-subgraph payroll system, the 4-legged identity, hybrid doc OCR, and the simulation sandbox every action-taking agent needs.
The single highest-leverage decision when shipping mission-critical autonomous agents. Production is the only truth.
Why five specialized $0.01 agents beat one $0.50 god model, and what the multi-agent crowd gets wrong about it.
Why human-in-the-loop is the only ethical and profitable way to scale agentic AI in a world of bot fatigue.
Agent Experience is the new SEO. Here is what it means, what changes, and the four-step audit to figure out how your product looks to the agents already using it.
The hidden engineering that decides whether your agent makes it to production. The 65/95 gap and the three foundations underneath it.
90% of enterprise data is locked in PDFs. The 2026 pipeline that gets it out is not RAG, not vision-only, and not the OCR you remember from 2018.
Privacy, security, and consent when agents have access to your terminal and your sensitive data. The 2026 framework, and the EU AI Act deadline most teams are sleeping on.
What the Model Context Protocol actually is, what it gets right, where it leaks, and why the local-first version is the cleaner story.
The engineering math behind preventing an agentic loop from burning through your monthly runway in one night.
Why agents forget by default, what the four types of memory actually are, and how to build a system that compounds across sessions.
OAuth was designed for three actors. Agentic systems have four. Here is what breaks, what RFC 8693 fixes, and why most teams are shipping shared credentials anyway.
Why bigger context windows are not the answer, and what production-tuned engineers actually trust in 2026.
The fastest 2026 teams are testing autonomous agents in synthetic enterprise environments before any customer is exposed. With the case for it and the open-source pieces to build one.
What agentic workflows are actually doing to entry-level engineering, and what to do about it.
How to lead a codebase by stating intent instead of writing syntax, and the discipline that keeps it from falling apart.
The two dominant agent reasoning patterns in 2026, what they get right, where each one fails, and how to know which to pick.
Pieter Levels at $420K a month. Marc Lou at $1M a year across twelve micro-SaaS. Tony Dinh at $1M working twenty hours a week. The narrow real pattern, with sources, costs, and where it breaks.
The build, the OpenClaw config, and the first agent worth running. End to end on a Framework 16 with 96GB unified memory.
Agentic loops that detect, diagnose, and fix deployment errors before you see the notification. With the workflow that actually works in 2026.
A framework for finding which 20% of your tasks are agent-ready before you write a line of code.
Using Apify, Firecrawl, and a local model to monitor every move your competitors make in real time. With the architecture and the weekly digest format that actually gets read.
A research-first outbound agent that scrapes news, LinkedIn, and financials before drafting an email. With the architecture, the prompts, and the guardrails.
Triage that does not just summarize. It prepares the drafts and fetches the data, and you approve. The 60-line config that actually works.