The Promise and Tradeoffs of Running AI on Phones and Other Devices

AI often brings to mind enormous cloud data centers, GPUs and energy consumption. I think more computation will move to devices close to the user, such as phones, laptops and cars. Local processing can reduce network dependence and keep some data on the device, though privacy, latency, energy and cost depend on the actual implementation and workload.

Qualcomm invited me to its AI event and partnered with me on this video. Its view of a future with more edge computing resembles mine, and I came away excited about running models on personal devices. The company also sent me a Galaxy S24 Ultra to try, an important context for the demonstrations I discuss.

Local AI could give people more control over their tools and reduce the need to send every request to a server. Qualcomm and other chipmakers are developing hardware designed for AI workloads. I expect small models to improve, but my estimate that they could address 90–95% of use cases is a prediction, not a measured universal coverage rate.

Rather than sending every prompt to a cloud model, an application could process suitable requests locally and route more demanding ones to a larger model. Whether that improves speed, energy use or price depends on the device, model, network and volume of requests; a cloud request is not inherently slower or more wasteful in every case.

Mixture-of-Agents research combines outputs from multiple models to produce a response. I describe this as a way for smaller agents to collaborate, but the reported results depend on the specific models, evaluation and additional computation used. They do not prove that any group of small models will match frontier systems or run on a phone.

RouteLLM explores routing queries between a stronger and weaker model to balance quality and cost. I characterize its findings as sending 90% of use cases to a small model and saving 80%, but the researchers' reported savings vary by benchmark: their 2024 report gives over 85% on MT Bench, 45% on MMLU and 35% on GSM8K at a stated quality target. Its experimental weaker model was Mixtral 8x7B, not evidence of an on-phone deployment or a universal task percentage.

I expect better compression, quantization and model research to make more capable local systems possible. That is a projection rather than a guarantee that the majority of real-world workloads will fit on any particular device.

At the Qualcomm event, I saw LM Studio, a tool I like for running models locally, demonstrated on Qualcomm hardware. The event also showed a car interface with an AI assistant and intelligent drones envisioned for personal use, rescue and delivery. I describe these as demonstrations, not independent evaluations of production reliability.

The event included Copilot+ PCs using Snapdragon X Elite chips and Qualcomm AI Hub tools intended to optimize models and deploy them in applications. The range of devices is the point: edge processing can mean a computer or vehicle as well as a phone.

The Galaxy S24 Ultra Qualcomm sent me uses a Snapdragon 8 Gen 3. Some Galaxy AI features rely on cloud services, while others can process information on the device. I highlight live call translation and Chat Assist for writing and translation, although processing modes and availability depend on the feature and configuration.

I also point to AI-assisted phone photography. The on-screen Generative Edit example is accompanied by a notice that it requires a network connection and Samsung account; it therefore should not be taken as evidence that every photo feature runs locally. The captions around this section are incomplete, so the visual notice helps clarify the demonstrated feature.

Looking ahead, I imagine personal agents working on a phone to arrange appointments, answer email, plan events and pay bills. These are examples of the future I hope for, not tasks demonstrated here. Access to personal information and the ability to take actions would also depend on permissions, software and safeguards.