Yesterday, we tested Grok 4.5 that had just come out. At first, we were disappointed. But then one of our engineers looked more closely.
I head the AI research department at XBOW, a cybersecurity company specializing in autonomous pentesting. Model choice is a big topic here, and when new models come out, we’re usually among the first to benchmark them on our cybersecurity benchmarks.
So when Grok 4.5 was pushed out, I was curious – its direct predecessor hadn’t performed particularly well, but the Grok series had had some strong contenders in the past, and “trained alongside Cursor”.... Well, it can’t hurt, can it. Especially an agent that needs to address fiddly code problems could well profit from it.
The numbers in our first test were a mixed bag: it seemed effective, but not efficient. Meaning it could solve most problems we gave it, but it spent a whole lot of money on the way. We might have discarded it right there, but something didn’t feel right, given that its per-token price actually sounded quite reasonable. Where was it spending the money?
Our agent traces looked normal. But Bjørn Christian Skov Jensen discovered an interesting anomaly: the model wasn’t failing the tasks – it was failing the calls. It would call out for a completion, but the returned string was unparseable, so it had to retry. Usually several times. So we paid double and triplicate for each turn of our agents. All because Grok really chafed under the yaml format we instructed it to give its answers in.
That didn’t degrade performance – whenever it messed up (i.e. often) and an immediate retry didn’t work, we’d instruct it with more details about its format errors, and eventually it would get it right. The funny thing was: these answers were actually really clever, it’s just that for the life of it the model couldn’t bring itself to obey our formatting instructions.
Of course, nowadays a model answer shouldn’t need to be parsed: tool calling should enforce the right format by itself. It’s just that we hadn’t used Grok in production for a while, and our Grok connector had gone a bit stale. So that option wasn’t available anymore, and we automatically fell back to our original way of talking to the models, and that was in yaml strings.
Once we updated the model backend, the test went swimmingly: Grok was just as clever, but now it also was also bang for buck.
At the end of the day, what remains is a lesson: AI has come a long way, but individual models still have individual quirks, systems can fail in unique and unexpected ways, and you can’t just run a new eval hands-off: you need to keep your eyes on the ball. Or the benchmark.
