Everyone and their dog sells AI Firewalls. And they’re not even firewalls!
How do you compare a SaaS one with an OOB API-based self-hosted container?
I’ve called out AI Firewalls for applying archaic architectures to LLM use cases.
User > Firewall > LLM > Firewall > User
Very 2000 and late
But there's more depth to these products. First, they're not firewalls because they don't filter packets. They host detection logic, so you have to re-learn how they fit in your env
I found these 4 dimensions to evaluate them against:
➡️ Stateful - The firewall maintains its own persistent, immutable record of the entire interaction context (prompts, responses, tool calls, API calls, reasoning traces) across turns and sessions, independent of and unaffected by any truncation or summarization applied to the LLM’s context window.
➡️Out-of-band - firewalls are obviously inline, but you also have an out-of-band little brother that is used for monitoring mode. The analysis verdict can then be implemented in your application.
➡️Vendor-hosted - detection logic runs on the vendor’s infrastructure. You can access that via:
1) API endpoints exposed by the vendor
2) An SDK, where a vendor library or framework hook embedded directly in your application or agent code
3) Through the CDN/SASE, where the vendor implements the firewall in the rest of their as-a-service network.
4) Proxy, where a non-CDN vendor hosts the firewall and you route your prompts through their firewall.
➡️Self-hosted - vendor gives you a form factor (container or VM) that runs the detection and enforcement engine to deploy in your prefered environment.
Let's see how these apply across vendors:
Statefulness
I wrote a paper for
AccuKnox about their stateful firewall, which is a very imporant capability for evaluating intent. I also found
Aiceberg can do full session-level analysis, but requires some configuration.
WitnessAI says the right words, but not enough to confirm.
As I was writing that paper, I saw that statefulness isn't always mandatory. LLMs are stateless so you always send the whole context. It matters once context gets summarized or truncated, because the firewall otherwise loses data the LLM no longer sees, which can contain malicious intent indicators.
Vendor hosted
Folks like
Cloudflare,
Cato Networks,
F5, and
Akamai Technologies run their own SASE/CDN, so they just put the firewall in their product. For an existing customer, this is a toggle switch and another line item on the invoice.
Self-hosted
HiddenLayer has some great docs that explain how this is architected. You can also look into
A10 Networks, Inc and
Check Point Software's
Lakera, now part of Check Point.
Palo Alto Networks is the most typical 'filter packets' firewall you'll see here, though detection runs in their cloud
Out-of-band
Nearly everybody does OOB verdict-only, because they can only enforce when in-line. There are OOB enforcement techniques like revoking tokens, but that's for another post