How to run local data inference inside a closed enterprise private network

The previous two lessons in this course covered a private RAG pipeline and getting a local runtime installed past a corporate proxy. This lesson is about the network layer underneath both: how you actually enforce "this machine cannot reach the public internet," and how you serve a local model to multiple internal tools once it's the only inference option available. Those are two separate, concrete engineering problems — a firewall policy someone can point to isn't the same thing as network isolation that's actually been tested.

Enforcing the Isolation, Not Just Assuming It

1. Block outbound by default at the OS/network layer. On Linux, this means an iptables OUTPUT chain with a default-drop policy and explicit allow rules for only what's needed — the model server's own loopback traffic, and nothing else.

2. Test the block, don't trust it. A closed network still needs DNS to resolve internal hostnames, and DNS queries are a well-documented way traffic leaks out of otherwise-isolated environments, since a sandbox that blocks direct TCP/UDP to external IPs may still permit outbound DNS lookups to arbitrary domains. Verify with an actual failed connection attempt from the isolated host, not just a policy document.

3. Give internal tools one place to talk to. Once cloud APIs are off the table, every internal script and tool that used to call OpenAI or Anthropic directly needs a new target. Standing up an OpenAI-compatible proxy in front of the local model(s) means those integrations change one base URL instead of being rewritten.

# 1. Allow loopback (the model server talking to itself / local clients)
sudo iptables -A INPUT -i lo -j ACCEPT
sudo iptables -A OUTPUT -o lo -j ACCEPT

# 2. Default-drop all other outbound traffic - nothing leaves unless
# explicitly allowed above this line
sudo iptables -P OUTPUT DROP

# 3. Confirm it actually holds - this should fail, not succeed
curl --max-time 5 https://api.openai.com

# 4. Serve the local model behind a single OpenAI-compatible proxy
# so internal tools point at one base_url instead of the model directly
litellm --config /etc/litellm/config.yaml
# config.yaml routes "my-model" -> openai/<local-model-name> at
# api_base: http://localhost:11434/v1 (the local Ollama/vLLM endpoint)

The order matters: isolate and verify first, then centralize how tools reach the model. Standing up a proxy on a network that hasn't actually had its egress tested just gives you a more convenient way to leak data, not a more secure one.