Gad Azeraf

The coding agent passed all tests -> Then we realized it had deleted the failing ones. Last semester, I supervised a student project exploring LLMs integrated into version control and CI/CD pipelines. Our question: If an autonomous coding agent can open PRs, review code, and make changes — will it stick to the rules? Or will it do the wrong thing? We did not know that this topic is so much timely right now. Turns out Spotify asked the same question. Their answer is instructive. As mentioned in the previous post, Spotify's agents run in strictly sandboxed environments: non-root containers (better is that compared to OpenClaw), limited command surface, no access to production. Essentially treating the agent like an untrusted but useful junior developer. The problem is not capability. It's reliability. Spotify explicitly reports failure modes like: – agents "fixing" failing tests by deleting or downgrading them – undoing a migration to make the build green again – producing PRs that pass CI but subtly violate the migration intent These aren't edge cases. These are predictable failure patterns. Something a junior developer would never do, right? This matches what I've observed firsthand — and what Karpathy describes from his own workflow: The mistakes aren't syntax errors anymore. They're subtle conceptual errors. The agent makes wrong assumptions and runs with them. It won't seek clarification, surface inconsistencies, or push back when it should. It will produce 1000 lines of bloat — and when you say "couldn't you just do X?" it cuts it to 100. But it won't do that unprompted. Spotify's answer is more subtle: a layered verification model: 1. Deterministic verification loops — build and test cycles with feedback wired back into the agent's reasoning before any PR is created. 2. "LLM as a judge" — a second model evaluates the diff against the migration spec. Did the agent implement the change, or merely silence symptoms? 3. Human review remains non-negotiable — especially for semantically rich changes. Even GitHub's Copilot agent defaults to human review over auto-merge. For organisations adopting coding agents in 2026, the strategic shift is this: Stop thinking in terms of AI features. Start thinking in terms of AI-enabled operating models for your codebase, your products. These tools are here to stay and evolve in unbelievable speed. How far would you let a coding agent go before a human must step in? #CodingAgents #SoftwareEngineering #AIGovernance #DevSecOps #TechLeadership #EngineeringManagement #RiskManagement

14