The Promise and Tradeoffs of Running AI on Phones and Other Devices

SUMMARY

In 30 seconds

Matthew Berman argues that smaller models and specialized chips could move more AI processing from cloud servers to personal devices. He reports on demonstrations at a Qualcomm event and discusses model-routing research and Galaxy S24 Ultra features. Qualcomm invited him, partnered on the video and supplied the phone; the benefits and future capabilities he describes depend on the model, device and application.

1A disclosed partnershipQualcomm invited Berman to its event, partnered on the video and provided an S24 Ultra. He presents a favorable view of the company's edge-AI direction.
2Smaller models and routingBerman discusses Mixture-of-Agents and RouteLLM as ways to improve or route model responses. The papers' results depend on the models and evaluations used.
3Event demonstrationsHe describes LM Studio, vehicle interfaces, drones, Snapdragon PCs and Qualcomm AI Hub as examples of local AI work.
4Phone features and limitationsHe highlights translation, writing and photography features. Some Galaxy AI functions use cloud services; the displayed Generative Edit footage explicitly requires a network connection.
5A future agent visionBerman imagines agents handling appointments, email and bills, but does not demonstrate those tasks operating autonomously on the phone.

Takeaways

Running suitable AI models locally can reduce data transfers and network dependence, but privacy, latency, power and cost depend on the implementation.
Berman disclosed a partnership with Qualcomm, its invitation to the event and the Galaxy S24 Ultra it provided.
RouteLLM's reported cost savings are benchmark-specific and do not establish that 90% of all tasks can run on phones.
Galaxy AI features can use on-device or cloud processing; the shown Generative Edit example states that a network connection is required.
Autonomous personal agents that handle bills and appointments are Berman's vision, not capabilities demonstrated in this video.

TopicsLocal inference Model routing & gateways AI cost & performance AI agents Privacy & data governance

FULL TEXT

Read the edited text by chapter

Show transcript

Transcript only

The case for edge AI

Berman contrasts cloud infrastructure with processing on personal devices. He discloses Qualcomm's invitation and partnership and discusses potential advantages and tradeoffs.

AI often brings to mind enormous cloud data centers, GPUs and energy consumption. I think more computation will move to devices close to the user, such as phones, laptops and cars. Local processing can reduce network dependence and keep some data on the device, though privacy, latency, energy and cost depend on the actual implementation and workload.

Qualcomm invited me to its AI event and partnered with me on this video. Its view of a future with more edge computing resembles mine, and I came away excited about running models on personal devices. The company also sent me a Galaxy S24 Ultra to try, an important context for the demonstrations I discuss.

Local AI could give people more control over their tools and reduce the need to send every request to a server. Qualcomm and other chipmakers are developing hardware designed for AI workloads. I expect small models to improve, but my estimate that they could address 90–95% of use cases is a prediction, not a measured universal coverage rate.

Rather than sending every prompt to a cloud model, an application could process suitable requests locally and route more demanding ones to a larger model. Whether that improves speed, energy use or price depends on the device, model, network and volume of requests; a cloud request is not inherently slower or more wasteful in every case.

Mixtures, routing and model research

He describes Mixture-of-Agents and RouteLLM as approaches for combining models or selecting between them. Their benchmark findings do not automatically generalize to all phone tasks.

Mixture-of-Agents research combines outputs from multiple models to produce a response. I describe this as a way for smaller agents to collaborate, but the reported results depend on the specific models, evaluation and additional computation used. They do not prove that any group of small models will match frontier systems or run on a phone.

SLIDE 1 / 3RouteLLM's routing diagramThe diagram routes a query to Mixtral 8x7B or GPT-4; it does not demonstrate either model running on a phone.

RouteLLM explores routing queries between a stronger and weaker model to balance quality and cost. I characterize its findings as sending 90% of use cases to a small model and saving 80%, but the researchers' reported savings vary by benchmark: their 2024 report gives over 85% on MT Bench, 45% on MMLU and 35% on GSM8K at a stated quality target. Its experimental weaker model was Mixtral 8x7B, not evidence of an on-phone deployment or a universal task percentage.

I expect better compression, quantization and model research to make more capable local systems possible. That is a projection rather than a guarantee that the majority of real-world workloads will fit on any particular device.

Qualcomm event demonstrations

Berman recounts demos of LM Studio, vehicle interfaces, drones and Snapdragon-powered PCs. He also mentions tools for optimizing and deploying models.

At the Qualcomm event, I saw LM Studio, a tool I like for running models locally, demonstrated on Qualcomm hardware. The event also showed a car interface with an AI assistant and intelligent drones envisioned for personal use, rescue and delivery. I describe these as demonstrations, not independent evaluations of production reliability.

SLIDE 2 / 3LM Studio at the Qualcomm eventThe event footage shows a local model interface displayed on a Qualcomm-powered computer.

The event included Copilot+ PCs using Snapdragon X Elite chips and Qualcomm AI Hub tools intended to optimize models and deploy them in applications. The range of devices is the point: edge processing can mean a computer or vehicle as well as a phone.

Galaxy AI and future personal agents

He discusses phone translation, writing and image features, and imagines agents handling personal tasks. The demonstrations and the future vision do not establish that all features run locally or autonomously.

The Galaxy S24 Ultra Qualcomm sent me uses a Snapdragon 8 Gen 3. Some Galaxy AI features rely on cloud services, while others can process information on the device. I highlight live call translation and Chat Assist for writing and translation, although processing modes and availability depend on the feature and configuration.

I also point to AI-assisted phone photography. The on-screen Generative Edit example is accompanied by a notice that it requires a network connection and Samsung account; it therefore should not be taken as evidence that every photo feature runs locally. The captions around this section are incomplete, so the visual notice helps clarify the demonstrated feature.

SLIDE 3 / 3Galaxy Generative Edit exampleThe on-screen notice says the illustrated feature requires a network connection and Samsung account.

Looking ahead, I imagine personal agents working on a phone to arrange appointments, answer email, plan events and pay bills. These are examples of the future I hope for, not tasks demonstrated here. Access to personal information and the ability to take actions would also depend on permissions, software and safeguards.