Is Local AI Actually Private? A Practical Threat Model for On-Device LLMs

Short answer: Running an LLM locally removes the biggest privacy risk — your prompts and documents never cross the network during inference. But "local" is not the same as "airtight." Some tools still send telemetry, every tool downloads its model over the internet at least once, and a badly configured RAG setup can quietly ship your data to a remote embedder or an unencrypted index. Privacy is a property you verify, not a checkbox you inherit.

What genuinely stays on-device

When inference runs locally, the data plane is genuinely local. Your prompts, your responses, the source documents you feed into retrieval, and the compute itself all stay on the machine. Apple's Core ML docs are blunt about the payoff: running strictly on-device "removes any need for a network connection," which also removes the class of risk that comes with one.

For RAG specifically, the parts that usually leak in cloud setups can stay put:

• Prompts and responses — never POSTed to an API.
• Documents — indexed in place, not uploaded.
• Embeddings and retrieval — AnythingLLM, for example, keeps embeddings, retrieval, and context all local.
• The context window — assembled on-device from local chunks.

Tools built local-first lean into this. GPT4All's LocalDocs runs fully offline with no telemetry, explicitly aimed at privacy-conscious users. That "no telemetry" claim is the part that matters — because it is not the default everywhere.

What still phones home

This is the honest gap. Local inference does not guarantee local everything:

• Telemetry and analytics. Many desktop AI apps ship usage analytics on by default. It is often anonymized and rarely includes prompt content — but it is still a network call you did not ask for.
• Update checks. Version pings on launch. Low-risk, but they reveal you are running the tool and when.
• Model downloads. Every tool fetches weights from Hugging Face or a mirror the first time. Nothing sensitive leaves, but it is a network dependency, and a compromised mirror is a supply-chain concern.
• Remote embedders. The subtle one. If your "local" RAG pipeline calls a hosted embedding API to vectorize documents, your documents are leaving — chunk by chunk — before they ever reach the local vector store.

Why on-device is the honest fit for regulated work

The use cases people cite for local LLMs — HIPAA-covered health data, privileged legal material, proprietary source code — are exactly the ones where network egress is the liability. Multiple 2025 deployment guides land on the same reasoning: on-device keeps prompts, documents, and sensitive data off the network, so there is no third-party processor and no data-in-transit to reason about.

But do not overclaim. On-device processing removes egress; it does not, by itself, make you HIPAA-compliant. Compliance still depends on full-disk encryption, access controls, and audit logging on the device. Local inference is a necessary condition, not a sufficient one.

Hardening checklist

If you actually care about the privacy claim, verify it:

• Block telemetry and update checks in the app's settings before first real use.
• Watch the network while the app runs. On macOS, Little Snitch flags every outbound connection; for a quick check, lsof -i or a short packet capture works. See exactly which hosts the process talks to.
• Confirm the embedder is local. If your embedding model name maps to a hosted API, your RAG is not local.
• Encrypt the on-device index. The vector store and document cache are plaintext on disk by default in most tools. Put them on an encrypted volume.

# See every network connection an app opens, live
sudo lsof -i -P | grep -i "gpt4all\|ollama\|anythingllm"
# Empty output during a query = inference is staying local

Where I draw the line in Ruma

In Ruma, my rule is that the default path is fully on-device — embeddings, index, retrieval, and inference — with zero telemetry, because that is the only version of the promise I can actually verify. Anything that touches the network is opt-in, named explicitly, and never the fallback that kicks in silently when a local model is slow. If a user turns on a cloud model, they chose it. The moment "local-first" quietly degrades to "cloud when convenient," the privacy claim is dead — so I would rather ship a slower honest default than a fast dishonest one.

Local inference kills the egress risk; telemetry, downloads, and RAG plumbing are where privacy actually leaks — so verify, don't assume.

Related: more writing.