What changes when your LLM runs on the user's machine

When you build against a hosted model API, you design around latency and rate limits. When you build against a local model through Ollama, the constraints flip. Latency is often lower and free, but you now have to account for wildly different hardware, models that may not be pulled yet, and no guarantee of availability if the user's machine is busy.

The frontend has to do more work as a result. Streaming stops being a nice-to-have and becomes required, because local inference on modest hardware can be slow enough that a non-streaming UI feels broken. You also need honest error states, like "model not found, run ollama pull llama3.2," because that is a real state the UI has to handle.

The payoff is a different privacy story: nothing leaves the device, which changes what you can promise the user and what you do not need to build. That is worth designing for on purpose, not drifting into by accident.