AI
This lesson closes the loop this course has been building toward: lesson 3 covered blocking outbound network access from a closed environment, but most organizations still need some employees or tools to reach a cloud LLM API — just not with each of them holding the organization's real provider key. A reverse proxy sitting between internal clients and the cloud API solves that: it's the one thing on the network actually authorized to hold the real OpenAI/Anthropic API key, and every internal caller gets an internal credential instead.
What The Proxy Is Actually Doing
1. Holding the real key. The provider's actual API key lives only in the proxy's server-side config or environment variables — it's never distributed to individual developer laptops, scripts, or client apps.
2. Issuing internal credentials instead. Each team or user gets a proxy-issued "virtual key." When a request arrives with a virtual key, the proxy authenticates it, then substitutes the real provider key before forwarding the request upstream — the client never sees or holds the credential that actually matters.
3. Enforcing limits per key. Because every request already passes through one chokepoint, that's also the natural place to attach per-team spend limits, rate limits, and model-access restrictions — a key that should only reach one internal model can be blocked from calling anything else.
# LiteLLM Proxy config - the real provider key is set as an
# environment variable on the SERVER only, never in a client app
model_list:
- model_name: internal-gpt
litellm_params:
model: openai/gpt-4.1
api_key: os.environ/OPENAI_API_KEY # real key, server-side only
# Start the proxy
litellm --config proxy_config.yaml
# Issue an internal "virtual key" to a team - this is what actually
# gets distributed to developers, not the real OPENAI_API_KEY
curl -X POST http://localhost:4000/key/generate \
-H "Authorization: Bearer $PROXY_MASTER_KEY" \
-d '{"models": ["internal-gpt"], "max_budget": 50}'
# Returns something like sk-litellm-... - THIS is what a client app uses| Without a Proxy | With an Internal Reverse Proxy |
|---|---|
| The real provider key is copied into every app/script that calls the API | The real key exists in exactly one place: the proxy's server-side config |
| Revoking access means rotating the key everywhere it was copied | Revoking access means deleting one virtual key at the proxy |
| No natural place to enforce per-team budgets or model restrictions | Budgets/model access are enforced per virtual key at the one chokepoint |
This is also the standard pattern NGINX's own engineering blog documents for AI-proxy deployments — provider credentials set in the proxy configuration and injected into upstream requests, never exposed to the calling client. LiteLLM Proxy and an NGINX-based setup are two implementations of the same underlying idea: isolate the credential, distribute internal tokens instead.
Practical Challenge
Run LiteLLM Proxy locally with a placeholder API key, generate two virtual keys with different max_budget values via the /key/generate endpoint, and confirm a request using one virtual key is rejected once its budget is exceeded — without ever putting the real provider key in the client request.
Concept Check
Sources & Further Reading
- LiteLLM Proxy — Virtual Keys — official docs for the virtual-key issuance, per-key budgets, and server-side credential isolation pattern used in this lesson.
- NGINX — Using NGINX as an AI Proxy — NGINX's own engineering blog confirming the same credential-isolation pattern (real keys held server-side, injected into upstream requests) implemented with NGINX/NJS instead of LiteLLM.