Local Inference on Desktops and Phones

Every lesson so far assumed that the model runs in a datacenter and the user's device sends requests to it. There is another option: run the model on the user's own laptop, desktop, or phone. This is called local inference. It is also called edge inference, client-side inference, or on-device inference. In all of these, the end user's device does the computing instead of a central server.

This lesson explains why that is attractive, why most inference still happens in the cloud, and what hardware and software exist for desktops and phones. Afterwards you can estimate whether a model fits a device and how fast it can decode, and you can decide which requests belong on the device and which belong in the cloud. The terms VRAM, bandwidth, and decode come from Inside a GPU.

Example 1: four advantages of running on the device

Client-side inference has four large advantages over server-side inference.

The cost row works because the user pays for the hardware and the electricity. The developer does not rent a GPU for each user.

A worked case on the latency row uses made-up round numbers, labeled as such. Suppose a cloud request has 120 ms of network travel in both directions, 20 ms waiting in a queue, and 30 ms of prompt processing before the first token. The time to the first token is 120 + 20 + 30 = 170 ms. A small on-device model with no network and no queue, and 90 ms of prompt processing on a modest chip, needs only 90 ms. The device wins even though its chip is much slower, because the fixed costs of the network disappeared. If the prompt were long enough that the device needed 400 ms, the cloud would win. The advantage is real and limited by the speed of the device, as the next example shows.

Example 2: four weaknesses

Local inference looks perfect in theory. In practice most inference happens in the cloud, for four reasons.

The audience matters. An AI enthusiast may have the newest phone and a fast computer. The median user is more likely to have older devices with weaker components. A feature that works on the developer's machine can be unusable for most customers.

Local inference is nonetheless moving from experiment to production. The hardware and software ecosystem is strong, and a lively community, closely tied to the world of open models, drives it.

Example 3: desktops, and why Apple leads

The classic local device is a workstation or gaming PC with one or two high-end consumer GPUs from NVIDIA or AMD. Researchers and enthusiasts use these setups, but they are a small part of the desktop inference market.

Apple is increasingly the leader. Apple's custom M-series CPUs and GPUs share a single pool of unified memory, meaning the CPU and the GPU read the same RAM. A GPU can therefore use far more memory than a discrete card has, although the memory is slower. The best options from Apple and NVIDIA show the trade-off between capacity and speed.

The trend continues in midrange hardware at more reasonable prices. Low-end computers such as Chromebooks cannot run any meaningful local inference.

A worked comparison of the two columns:

memory per $1,000:  RTX 5090: 32 GB / 5 = 6.4 GB      M3 Ultra: 512 GB / 10 = 51.2 GB
memory ratio:       512 / 32 = 16x more memory on the M3 Ultra
bandwidth ratio:    1,792 / 819 = 2.19x more bandwidth on the RTX 5090

The M3 Ultra holds 16 times as much and the RTX 5090 moves data 2.19 times as fast. Capacity decides which models can load at all, and bandwidth decides how fast they decode.

Example 4: what fits and how fast it decodes

Inside a GPU gave two rules. The weights need parameters x bytes per parameter of memory, plus at least 50 percent headroom for the KV cache. A rough ceiling on decode speed for one request is bandwidth / bytes read per token. The numbers below apply them to the two desktops. A 4-bit model uses half a byte per parameter (Quantization covers formats). Every model here is hypothetical, and the speeds are ceilings.

Dense 8B model, 4-bit: weights = 8 x 0.5 = 4 GB, needed = 4 x 1.5 = 6 GB
  RTX 5090 (32 GB): fits.  ceiling = 1,792 / 4 = 448 tokens per second
  M3 Ultra (512 GB): fits. ceiling =   819 / 4 = about 205 tokens per second

Dense 70B model, 4-bit: weights = 70 x 0.5 = 35 GB, needed = 35 x 1.5 = 52.5 GB
  RTX 5090: does not fit (52.5 > 32, and even the weights alone are 35 > 32)
  M3 Ultra: fits.  ceiling = 819 / 35 = about 23 tokens per second

MoE with 120B total and 5B active parameters, 4-bit: weights = 120 x 0.5 = 60 GB, needed = 90 GB
  RTX 5090: does not fit
  M3 Ultra: fits.  bytes read per token = 5 x 0.5 = 2.5 GB, ceiling = 819 / 2.5 = about 328 tokens per second

The 8B model decodes faster on the discrete GPU because its bandwidth is higher. The 70B model runs only on the machine with large memory. The third case shows why Mixture of Experts (MoE) is a tailwind for desktops. An MoE model has many "expert" sub-networks and activates only a few for each token (Attention and Mixture of Experts). Its active parameters are a fraction of the total, so one user's single request touches only a fraction of the weights. The model needs the memory for all the weights, and the speed depends on the active ones. A large-memory machine stores the whole model and reads little of it per token. The ceiling simplifies reality, since different tokens pick different experts and the KV cache also uses bandwidth.

Example 5: the desktop software ecosystem

The open-source community, led by enthusiasts, focuses on running frontier open models on desktops and laptops. Today it is possible to run aggressively quantized models with more than 100 billion parameters on high-end personal hardware using tools such as Ollama and llama.cpp. A quantized model stores each weight in few bits, as in the 4-bit cases above.

The same machines run other modalities:

• Image generation is popular on personal computers, especially through ComfyUI, a tool for assembling several image-model components into one pipeline.
• Smaller language models and speech are feasible on midrange computers.
• Browser inference libraries such as WebLLM and other cross-platform standards are being developed quickly, to bring these abilities from early adopters to the mainstream.

Example 6: phones

On-device inference on mobile devices accounts for the majority of on-device workloads today. Both major operating systems provide tools for developers.

Phones have very limited hardware and battery capacity, so inference is harder than on a desktop. Even high-end phones struggle with models that have more than one or two billion parameters. The same rules give the numbers:

1B parameters at 4-bit:  1 x 0.5 = 0.5 GB of weights
2B parameters at 4-bit:  2 x 0.5 = 1.0 GB of weights
2B parameters at 16-bit: 2 x 2   = 4.0 GB of weights

A 70B model at 4 bits needs 35 GB, which is far beyond any phone, so the one-to-two-billion limit is a hard practical boundary.

Some modalities suit phones well:

• Transcription (speech to text) and speech synthesis (text to speech) are latency-sensitive, and some models are small enough to run in real time on modern phones.
• Discrete tasks such as translation can be handled on the device by small fine-tuned models.

Example 7: local and cloud together

The future of inference is not local or cloud. It is both working together to give fast, smooth experiences. Small models and quick queries run on the user's device, and more demanding workloads stay on datacenter GPUs in the cloud. A router that applies this idea can use simple rules, shown here as a decision table with made-up rules.

flowchart TD
  R["Request"] --> Q{"Small model, private data, offline, or latency-sensitive?"}
  Q -- "yes" --> D["Run on the device"]
  Q -- "no" --> W{"Weak device, low battery, or large demanding model?"}
  W -- "yes" --> C["Run in the cloud"]
  W -- "no" --> D2["Run on the device"]

The diagram shows the order of the checks. The table lists the cases in detail.

Interview points

• Local inference runs the model on the end user's device. Its advantages are zero network latency, independence from connection and server load, better privacy, and no GPU cost for the developer.
• Its weaknesses are limited hardware, thermal constraints, a fragmented hardware and software support matrix, and battery drain. The median user has older, weaker devices than an enthusiast.
• Apple M-series chips use unified memory. The M3 Ultra offers 512 GB at 819 GB/s for about $10,000. The RTX 5090 offers 32 GB at 1,792 GB/s for about $5,000. Capacity decides what loads and bandwidth decides decode speed.
• Aggressively quantized 100B+ parameter models run on high-end personal hardware with Ollama and llama.cpp. MoE models help because one request touches only a fraction of the weights.
• ComfyUI runs image generation pipelines, and WebLLM runs models in the browser.
• Phones rely on Android's AI Edge SDK and ML Kit GenAI (Gemini Nano, Gemma) and iOS's Foundation Models and Core ML. Models above one or two billion parameters are hard on phones.
• Transcription and speech synthesis suit phones because they are latency-sensitive. Translation suits small fine-tuned models.
• The practical design is hybrid: small models and quick queries on the device, demanding workloads in the cloud.

Summary

• Local inference is also called edge, client-side, or on-device inference.
• Advantages: zero network latency, independence, privacy, and cost.
• Weaknesses: hardware capability, thermal limits, a fragmented support matrix, and battery life.
• Desktop: NVIDIA RTX 5090 (32 GB, 1,792 GB/s, $5,000) against Apple M3 Ultra (512 GB, 819 GB/s, $10,000). Chromebooks cannot run meaningful local inference.
• Tools: Ollama, llama.cpp, ComfyUI, WebLLM.
• Mobile: Android AI Edge SDK, ML Kit GenAI, Gemini Nano, Gemma. iOS Foundation Models and Core ML. A practical limit of one or two billion parameters.
• Local and cloud inference work together.