How to Audit a Local AI App's Privacy

A local AI privacy audit should trace each user action from input to output. The label “local” is useful only when you can identify where inference runs, where data is stored, and which actions cross the network.

Use the process below before placing sensitive material in an AI app. It works for browser tools, desktop applications, and mobile apps.

Separate inference from storage

Start with two independent questions:

1. Where does the model process the prompt?
2. Where does the application save the conversation?

An application can store history on your device while sending prompts to a hosted API. It can also run AI inference locally while sending analytics or connected-tool requests elsewhere. Neither storage location nor account requirements prove where generation happens.

Look for a named runtime, a model artifact you can download or import, and a clear statement that ordinary prompts and responses are processed on the device. Google AI Edge Gallery provides a useful reference implementation: its project documentation identifies an on-device runtime, local model management, and offline inference after a model is available.

Build an action inventory

List every action that could use the network. A typical local AI product has more than one path:

Action	Expected location	Questions to verify
Normal chat	Device	Does the prompt reach any remote inference endpoint?
Conversation history	Device	Which database or file store holds it?
Model download	Network, then device	Which host receives the request and where is the artifact stored?
Application update	Network	What metadata reaches the delivery host?
Web search	External service	Is the query shown before it is sent?
Feedback or report	Product server	Does the app preview the exact excerpts and metadata?
Analytics	Analytics service	Can events contain prompts, filenames, or generated text?
Crash diagnostics	Product or third party	Is it enabled by default and can logs contain content?

This inventory prevents an offline chat test from being treated as proof for unrelated features.

Test the normal path offline

Complete setup while connected, including the application shell, runtime components, and a compatible model. Then disconnect the device and start a new chat.

Verify that the application can reopen, load the selected model, accept a prompt, and produce a response. Test a fresh conversation and a saved conversation. If the app supports local files, attach a small document and confirm that extraction, retrieval, and citations still work without a connection.

An offline test is strong evidence for the workflow you exercised. It does not prove that every feature is local, and it does not reveal background requests made while the network is available. Use browser developer tools, an operating-system network monitor, or a controlled proxy for the connected test.

If you are still choosing a runtime, the local LLM setup guide explains the hardware and model tradeoffs that affect this test. Quantization is especially relevant because model size determines whether an artifact can fit in device memory and storage.

Inspect browser storage limits

Browser-based local AI commonly uses Cache Storage for application assets and IndexedDB for structured records or model artifacts. The IndexedDB API supports significant amounts of structured data, including files and blobs, and follows the browser’s same-origin policy.

Local browser data is still controlled by the browser and the user. Storage quotas and eviction behavior vary by engine. MDN’s storage quota guidance explains the difference between best-effort and persistent storage.

Check whether the app:

• requests persistent storage where appropriate;
• reports the size of downloaded models;
• handles quota failures without losing unrelated records;
• provides exports for important conversations or projects;
• explains what clearing site data will remove;
• deletes model files, indexes, attachments, and derived records together.

No cloud recovery is a real privacy benefit and a real durability tradeoff. The product should state both.

Verify encryption claims precisely

“Encrypted” needs a scope. Ask which records are encrypted, where the key lives, when the vault is unlocked, and whether exports or backups receive separate protection.

On Android, the Android Keystore can keep key material non-exportable and enforce restrictions outside the application process. Supported devices can bind a key to secure hardware. The presence of a Keystore key does not show that every sensitive record is encrypted or that plaintext is protected while the app is open.

For a browser vault, check whether the passphrase is sent to a server, whether the provider can recover it, and whether the app protects both structured records and file blobs. A product that cannot recover a local passphrase should make that consequence visible before the user depends on it.

Review every optional network action

Connected features should change the interface before they change the data boundary.

For web search, verify the provider, query, and confirmation step. For feedback or response reports, verify the exact excerpts, optional comment, product metadata, retention period, and deletion route. For analytics, verify the event names and prohibited fields rather than relying on a broad statement such as “anonymous usage data.”

A useful confirmation answers four questions:

1. What is being sent?
2. Who receives it?
3. Why is it required?
4. What happens after submission?

Reject designs that send content before presenting consent or hide a connected action behind a generic icon.

Run a repeatable audit

Record the app version, platform, browser or operating system, model, and date. Then run the same sequence in a clean profile or test device:

1. Install or open the application.
2. Download one model and record the delivery hosts.
3. Start a normal chat while monitoring requests.
4. Disconnect and repeat the chat.
5. Import a file and repeat the request check.
6. Trigger each optional search, report, analytics, and diagnostic control separately.
7. Export and delete the conversation, project, file, index, and model.
8. Reopen the app and verify what remains.

CuriousLM publishes a 25-question private AI checklist that expands this process across accounts, retention, permissions, backups, citations, and high-risk use. It also documents a concrete boundary: normal inference and private workspace data stay on the device, while delivery, model downloads, updates, optional confirmed search and reports, and disclosed aggregate analytics can use the network.

Write the result as an action map, not a single privacy score. A failed offline chat test, an undisclosed analytics event, and a missing deletion control are different findings with different fixes. The audit is complete when another person can repeat each test and observe the same boundary.