Secret Guard and local LLM gateways: protecting secrets while your agent uses them

An agent that does real work doesn't just answer questions: it calls tools, opens connections, queries databases. And to do that it needs secrets — API keys, tokens, credentials. The problem is that secrets enter the agent's context, and from there they end up in the prompt sent to the model.

Picture an employee you hand the safe key to, then leave working in the corridor: the key stays in their pockets, visible to anyone walking past. A local gateway does the opposite: the secret stays on your hardware, and the model only receives a dummy copy that is worthless outside that door.

How the redact works

Secret Guard (https://github.com/luochen1990/secret-guard) is a Rust gateway, MIT license, that sits between your agent and the provider. Two steps, one per direction:

1. Outbound redact: when a request passes through the gateway, every secret value it finds is replaced with a realistic mock: same length, same character set. To the model, a 32-character API token looks like a 32-character API token — the prompt doesn't break, and the structure that preserves the provider's prefix caching stays intact.
2. Inbound restore: the response passes back through the gateway, which maps the mocks to their real values. If a tool call carries a mock in its arguments, the same happens there: the agent receives the usable value without the model ever having seen it.

The mock map is deterministic and one-to-one: the same secret always produces the same mock, and different mocks never collide. That is the property that preserves the prefix cache — if mocks changed at random, every request would restart from zero.

The four per-secret strategies

Each registered secret has its own policy, and this is the point: there is no "hide everything", there is a choice per use case.

Strategy	Behavior	When to use it
redact/restore (default)	Replaces outbound, restores inbound	The tool needs the real value
reject	Returns 503 if the secret is present in the request	That secret must never leave, even masked
placeholder	Replaces with [REDACTED:{id}] and fires a canary alert if it reappears	You only want to know whether and where it circulates
monitor	Passes through, logs the crossings	Auditing, without changing behavior

Supported protocols

The gateway forwards OpenAI Chat Completions, Anthropic, Gemini, Ollama and Responses. Full redact/restore works on the OpenAI, Anthropic and Responses protocols, including cross-protocol translation (streaming included). Gemini and Ollama are currently byte-exact pass-through: redaction for them is on the roadmap, and if an unknown secret were to pass in clear text anyway, the gateway answers 503 — the failure is closed, not silent.

Five-step setup

cargo install --git https://github.com/luochen1990/secret-guard
cd /opt/secret-guard
secret-guard register --name "stripe" --value "sk_live_xxx"
secret-guard run --port 18787

1. Install the binary (rustc 1.96+ required; there is also a NixOS module services.secret-guard).
2. Register your secrets with secret-guard register: they land in secret-guard.state.toml, in plain text. Put it where you protect it: it is the file to gitignore and to back up encrypted.
3. Start the gateway: secret-guard run --port 18787. The management web UI is at http://127.0.0.1:8787.
4. Point your client at the gateway: for Claude Code, set ANTHROPIC_BASE_URL=http://127.0.0.1:18787 and a dummy ANTHROPIC_AUTH_TOKEN — the gateway forwards with its own credentials.
5. Verify: ask the model to repeat a registered key. If it works, it answers with the mock — the real value is not in the context.

If it doesn't work: the first thing to check is that the client actually points at the gateway (a curl to port 18787), then that the registry contains the secret — an unregistered secret is never intercepted, and the optional suspect scanner (never on by default) only flags suspicious values, never registering them on its own.

And the limits?

Be honest about what this covers and what it doesn't:

• The model never sees the real value, but it can try to get it out via prompt injection or a badly constructed tool call: the reject and placeholder strategies exist exactly for those cases, at the cost of a block.
• The state file is plain text: protection depends on where you put it, not on the gateway.
• It is a v0.1.0: the critical properties (byte-exact pass-through, reversibility, determinism, no collisions, multi-provider routing) are covered by property-based and end-to-end tests, but real-provider validation for Anthropic, Gemini and Ollama is declared as still to do on the roadmap.
• Per-secret configuration: there is no auto-protection; you declare what is a secret, and for whatever you don't know to be one, the suspect scanner is the only sentinel.

The comparison with the alternatives (LiteLLM, LLM Guard, PasteGuard) is in the repository, in docs/comparison.md, across seven dimensions.

The pattern to remember

The idea is not a longer password: it is a boundary. The secret lives on your hardware, the model sees only a realistic shadow, and the real value crosses the boundary in both directions without ever stopping in the middle. If your agents touch real data today — customers, orders, documents — that boundary is worth more than a good written policy.

To understand what actually goes over the network when AI is local, read Privacy: what actually goes over the network; and for the wider picture on agents that work locally, ML agents that run locally.

Sources

• Secret Guard — GitHub
• Comparison with LiteLLM, LLM Guard, PasteGuard

In summary

Secret Guard is a local Rust gateway that intercepts your agent's requests and swaps secrets for realistic mocks before they reach the provider, then maps them back to their real values in the response. Each secret has its own strategy (redact, reject, placeholder, monitor); the OpenAI, Anthropic and Responses protocols have full redaction, and Gemini and Ollama are in pass-through for now. The model never sees the real values, the prefix cache stays intact, and failure is closed (503) instead of silent. It is a v0.1.0 with the limits that come with it, but the pattern — a boundary between the agent and the secrets — is what a business running agents on real data should have from day one.

Article generated with AI