SHRIKESelf-Hosted LLM Security: The Two Halves
Self-hosted LLM security has two halves. The first is the serving engine: keep Ollama, vLLM, or LM Studio off the public internet, patched, and behind authentication. The second is the agent behind the endpoint: a local model ships with no provider guardrails, so the policy check, the scope, human approval, and the evidence record have to be supplied by the deployer.
Almost everything written about securing a local model covers the first half and stops. That is not a criticism; the first half is real and the guides that cover it are good. But a team that finishes the checklist and declares the model secured has secured the engine. The thing that reads the engine's output and acts on it is where the consequences live, and on a local stack it was never protected in the first place.
What does the first half look like?
The serving engine is ordinary infrastructure and fails in ordinary ways. Three items cover most of it:
- Exposure. Ollama's default configuration listens on all interfaces on port 11434 with no authentication. Internet-wide scans have counted about 175,000 publicly reachable instances (Sysdig, June 2026, via the Cloud Security Alliance). Bind to localhost or a private interface, and put authentication in front of anything that must be reachable.
- Patching. CVE-2026-7482 in Ollama, rated CVSS 9.1, allowed an unauthenticated caller to read process memory, including prompts, system prompts, and environment variables, from exposed servers; the reporting researchers noted the fix shipped without being flagged as a security patch (Cyera). vLLM and llama.cpp have carried their own advisories this year. Track the runtime like any other server component.
- Isolation. Run the engine as a non-root user with a read-only filesystem and dropped capabilities, and treat shared GPUs as a tenancy boundary (AccuKnox, SitePoint).
Stolen local compute is now used as attack infrastructure in its own right (see LLMjacking), which makes exposure a liability beyond the data on the box. The checklist ends when the engine is private, current, and isolated.
What is the second half?
A hosted provider runs controls beside its model that most teams never configured and rarely saw: a content safety pass, usage policies enforced at the API, and on the larger platforms an optional approval step and a log. They are tuned for the provider's liability rather than yours, but they exist. Open-weight models served locally ship with none of them; content filtering has to be added deliberately (Datanorth).
The consequences do not live in the model. They live in the actions an agent takes with the completion: the command it runs, the file it writes, the query it sends, the message it forwards. A local model that reads a poisoned document acts on it exactly as a hosted one would. The hosted path had a screen somewhere between the instruction and the act. The local path has an empty space. Filling it means supplying, at the boundary the deployer owns:
| Control | What it decides | Who supplied it on a hosted API |
|---|---|---|
| Policy check before the action | Is this input, tool call, or response hostile or leaking? | The provider, partially, for its own purposes |
| Scope | Is this action inside what the agent is for? | Nobody |
| Human approval | Should a person decide this one? | Some platforms, optionally |
| Session memory | Is this turn part of a pattern across the session? | Nobody |
| Evidence record | What was attempted, decided, and by whom? | The provider's log, on the provider's terms |
Do open-source guardrails cover it?
The content half, yes, and well. A common self-hosted stack runs Prompt Guard as a fast first gate, Llama Guard for hazard classification, NeMo Guardrails for dialog and execution rails, LLM Guard as a scanner, and Guardrails AI for output validation (Particula, Spheron). Each is good at its job, and a team with the headcount can operate all five.
What that stack does not contain is the action half of the table above: a declared scope, a hold that routes a consequential action to a person, memory across the session, and a per-action record in a shape an auditor accepts. Those are not classifier problems. They are governance problems, and they are the same on a local model as on a hosted one.
Where did the security work go?
It moved payrolls. With a hosted API the provider patches the engine and runs its own rails. Go local and both jobs belong to the in-house team, plus the rails the agent never had (Baytech names the lifecycle side of this tax). For a regulated organization that is often the right trade. It is only a mistake when it is made by accident, when the migration ticket said "data residency" and nobody wrote down that the guardrails were part of what left.
Related reading
- What is action governance? The pillar definition this page hangs off.
- When does an AI agent need governance? The five trigger moments, none of which depend on where the model runs.
- The stack you run when you go local: the bill of materials, row by row.
- Local models: governing Ollama, vLLM, or LM Studio through one base URL.
Shrike supplies the second half for a local model the same way it does for a hosted one: policy enforced on every tool call, command, and query before it runs, declared scopes, human approval, and a per-action evidence record, at the SDK, MCP, or proxy boundary you own. As a subscription that starts free, or deployed inside your boundary under an Enterprise engagement. See the local models guide.