Your Local Model Shipped Without Guardrails

Somewhere in your company this quarter, a team moved an agent from a hosted model to a local one. The reasons were good: the data stays on hardware you control, the bill becomes predictable, and nobody can deprecate the model out from under a workflow that depends on it. The migration took an afternoon. It also removed every provider-side control the agent had, in the same afternoon, and nothing in the migration mentioned it.

What a hosted model gives you without asking

A hosted provider runs controls beside the model that most teams never configured and never saw: a content safety pass on the way in and the way out, usage policies enforced at the API, and, for the larger platforms, an optional approval step and a log of what the model was asked to do. They are not strong controls. They are the provider's controls, tuned for the provider's liability rather than yours. But they exist, and the agent's behavior in production was quietly shaped by them.

A local runtime ships none of that. Ollama, vLLM, and LM Studio are excellent at one job: serving a model on an OpenAI-compatible endpoint. Content safety is not their job. An approval step is not their job. A record of what the agent did with the completion is not their job. The model behind the endpoint is often open-weight and lightly aligned, which is part of why it is cheap and fast. Whatever the hosted provider was doing for you is now being done by nobody.

The gap is not the model

It is tempting to read this as a model-quality problem and answer it with a better local model. That misses where the risk lives. An agent's damage arrives through its actions: the command it runs, the file it writes, the query it sends, the message it forwards. A local model that reads a poisoned document will act on the instructions inside it exactly as a hosted one would. The difference is that the hosted path had a screen somewhere between the instruction and the act, and the local path has an empty space.

The controls a local model needs are the controls any agent needs, and none of them belong inside the model:

• A policy check before the action. Prompt injection, data leakage, and unsafe commands caught before the agent executes, not after.
• A scope. What the agent is for, which tools it may use, and for how long. An action outside the scope is held for a person.
• Human approval on consequential actions. A local model has no vendor to route the decision to.
• An evidence record. Every allow, warn, hold, and block, on a record an auditor can read and a SIEM can ingest.

One base URL

Because those controls sit at the boundary the deployer owns, the SDK, the MCP server, or the proxy, they do not care which model is behind the endpoint. The Shrike SDK wrapper for OpenAI-compatible clients forwards a base URL to the provider client, so governing a local model is the same integration as governing a hosted one, with one line changed:

from shrike_guard import ShrikeOpenAI

client = ShrikeOpenAI(
    api_key="ollama",                        # local servers ignore the value
    base_url="http://localhost:11434/v1",    # the model endpoint: local
    shrike_api_key=os.environ["SHRIKE_API_KEY"],
)

response = client.chat.completions.create(
    model="llama3",
    messages=[{"role": "user", "content": "Summarize this ticket..."}],
)

TypeScript takes the same base URL through the provider options and Go through the client config; the local models guide has all three. Two endpoints are in play and it is worth keeping them apart in your head. The model call stays between your process and your runtime. The scan reaches the governance backend, which stores no content for safe interactions and keeps violation records encrypted for audit. If the detection pipeline itself has to run inside your boundary, that is an air-gapped deployment, and it is a different conversation from the one this post is about.

The property to check before you adopt any of this

The local-model migration exposes a question that is worth asking of every governance layer, including ours: where do the controls live? If they live inside the provider, they leave when the provider does, and every model change is also a controls reset. If they live at the boundary you own, a change of provider is a change of one line, the policy that held yesterday holds today, and the evidence record continues without a gap. Model choice stays yours. The controls, and the record they write, stay yours too.

That is the whole argument. A local model is a sound engineering decision that arrives with an empty space where the guardrails used to be. Fill the space with controls that belong to you, and the next migration, in whichever direction, costs one line. Start with the local models guide, then declare the agent's scope. Where this fits the larger picture is in what is action governance and the enforcement boundary is a deployment choice.