A browser can run private AI search, but the device decides how much

Jason Mayes explains local document search, model sizing and when a browser AI app should fall back to the cloud.

JMJason Mayesゲスト
チャンネル
Google Cloud Tech
公開日
2026年11月20日
動画
27:24
読了時間
9 分
原文
英語
YouTube で開く
スクロールすると動画は右下に小さくなって再生を続けます。字幕は閲覧言語に合わせて切り替わります。
閲覧言語2 言語で読めます
この記事はまだ日本語に翻訳されていないため、英語の原文を表示しています。
要約

30 秒でわかる

A browser can search private documents and generate answers locally, avoiding per-token cloud charges once the models are available. Web AI developer Jason Mayes says his local search library can scan 100,000 paragraphs in roughly 100 milliseconds. The challenge is that each user’s device has different capabilities: he recommends testing hardware, starting with the smallest model that meets the task, and using cloud models for demands the device cannot handle. A proposed shared browser cache could reduce repeat downloads of large models.

  1. 1A browser can search a private document collection locallyMayes says his library searches 100,000 paragraphs in about 100 milliseconds using the device’s graphics processor. A local language model can use the retrieved text to answer questions without a per-token cloud charge.
  2. 2An application must adapt to hardware it cannot chooseA browser may run on anything from a phone to a laptop. Mayes suggests cached shopping results and a local model for offline use, with image search on more capable devices.
  3. 3A tuned small model can handle a specialized taskMayes recommends testing a base model, gathering examples of its failures and retraining on those cases. This can avoid moving to a larger model for a few remaining errors.
  4. 4A proposed cache could prevent duplicate model downloadsCross-origin storage would allow sites to reuse an identical model already downloaded by another site, identified by a hash. Mayes presents it as a proposal.
  5. 5Benchmark the device before choosing local executionIf a quick performance test fails, Mayes suggests using the cloud. A capable device could use the cloud during the download, subject to the application’s privacy requirements.
  6. 6Start small and compare against an existing baselineChoose the smallest model that meets the task’s requirements, then consider fine-tuning for remaining failures. Mayes says an AI replacement should improve on the previous system’s measured performance.
  7. 7Routing needs both a speed test and a complexity testMayes proposes benchmarking a model architecture with placeholder weights before downloading the real model. He also suggests training a small classifier to route queries between local and cloud models.

ポイント

  1. Test the smallest model against the actual task before trying a larger one.
  2. Keep cached data and a preloaded local model available if offline operation matters.
  3. Measure device speed before downloading a model and route work with privacy requirements in mind.
  4. Fine-tune a near-miss model using examples of its failures.
  5. Compare the new system with a measured baseline, including the previous non-AI approach.
全文

チャプターごとに整えた全文を読む

文字起こしのみ
01

A full retrieval system can run in a browser

Mayes describes local search and generation that can keep documents on the device. His example searches 100,000 paragraphs in roughly 100 milliseconds.

AI applications have to balance speed, privacy, security and cost. Jason Mayes describes Web AI as running machine-learning models on the user's device inside a browser, using technologies such as WebGPU and WebAssembly. An emerging interface called WebNN is intended to give web applications access to dedicated neural processors.

These need not be tiny models, Mayes says: he has run a 31-billion-parameter Gemma 4 model locally when the device has enough graphics memory. Asked for a concrete example, he describes vector-search.js, a library he built to search information on the device.

His library uses EmbeddingGemma to turn text into numerical representations of meaning and searches them on the graphics processor. In his example, a PDF can be divided into paragraphs; he says he can search 100,000 paragraphs in roughly 100 milliseconds without sending the documents away. He sees applications in fields such as medicine, law and finance.

Aja Hammerly checks her understanding: an embedding model processes a collection of documents, creating a searchable database that a user can query in ordinary language. Mayes says the library compares the query with the stored representations; a local Gemma model can then use the retrieved passages to formulate an answer.

That combination makes a complete retrieval-augmented generation system possible in a browser, the two say. Once the necessary models and data are available locally, the search and answer can run offline, with the data staying on the device and no per-token cloud charge. Mayes adds that the strongest cloud models still have their uses.

02

Design for devices you cannot predict

Browser applications encounter a wider range of hardware than a fixed cloud deployment. Cached product search and image-based shopping illustrate fallback and added capabilities.

The difficult part is accommodating hardware the developer cannot predict. A cloud developer can choose a machine and know its memory and graphics hardware; a browser application may open on a watch, television, laptop, tablet or phone. Mayes says web developers must plan both for capabilities that are missing and for opportunities to add features on stronger devices.

For an offline train journey, a hybrid application might fall back from a cloud model to a smaller model already loaded on the device. It can still perform tasks within that model's limits rather than stop working altogether. Conversely, a device with more graphics memory might run a larger model and gain extra capabilities.

Hammerly asks what that would look like in a pet store. Mayes suggests caching products a returning customer previously searched for in a local database. Online, the store can draw on current cloud information; without a connection, the customer could still browse the cached subset, get recommendations and add items to a cart for later synchronization and checkout.

For a stronger device, the store could offer image-based search: photograph a collar or favorite toy and find visually and semantically similar products. A less capable device might use a text-only model and ask the customer to type a description instead. The more capable version makes the interaction easier without making it the only way to search.

03

Improve a small model on the cases it misses

Fine-tuning or distillation can adapt a model to a narrow task. Mayes outlines testing a base model, collecting failures and converting the retrained model for the browser.

A small local model may also improve at a particular task through fine-tuning or by distilling behavior from a larger model. Mayes says a specialized small model can sometimes match or exceed a larger one on that task, although producing it requires additional work and suitable data.

His proposed workflow starts by selecting a model architecture and size and testing its unmodified performance on the task. Developers then identify cases it gets wrong, gather examples—perhaps thousands from customers or another relevant source—and retrain the model. The retrained model can be converted for use with the browser runtime.

04

Use browser runtimes to execute models locally

LiteRT runtimes distribute model operations across supported local execution engines. Mayes then describes the tradeoffs among Gemma 4 model sizes.

Hammerly asks what LiteRT actually does. Mayes distinguishes LiteRT.js for general machine-learning models, including embedding models, from LiteRT-LM.js for language models. These runtimes load model weights and operations and distribute supported calculations to execution backends such as WebAssembly and WebGPU; here, backend means the local execution engine, not a cloud server.

The runtime can be imported like another JavaScript library. Mayes says a developer can load, configure and run a Gemma 4 model in fewer than 20 lines, provided the runtime supports the operations the model needs. At the time of the conversation, he says, the language-model runtime focuses heavily on Gemma.

Model size affects which devices can use an application. Mayes says the two-billion-parameter Gemma 4 model can run on lower-end devices, including older phones and devices such as a Raspberry Pi. He suggests the four-billion-parameter model for a more capable phone and estimates from memory that it uses around 4.5 GB of graphics memory.

According to Mayes, the two- and four-billion models accept audio, images and text, whereas the 12-billion model and larger versions handle images and text but not audio. He does not know the design team's reason, though he speculates that smaller models are aimed at devices where real-time voice interaction is useful. Other models could provide speech features on larger devices.

Mayes describes 12 billion parameters as a middle ground that may still struggle with tasks such as coding. He says the 26-billion-parameter mixture-of-experts model activates four billion parameters at a time, seeking a smaller model's speed with a larger model's knowledge. The 31-billion version is the largest in the family he describes.

05

Large model downloads need a caching strategy

Model files can be substantial and repeated downloads across sites are costly. Mayes describes a proposed shared storage standard for identical models.

Hammerly asks how a website packages something as large as a model. Mayes acknowledges the download can be substantial, but argues that a one-time cost can be worthwhile for a feature used repeatedly. Traditionally, separate web domains would each download their own copy, even if the model weights were identical.

Mayes describes a proposed cross-origin storage standard that would let one site request a model already downloaded by another using a hash identifying the exact file. The browser could then share the cached model rather than download it again. The discussion presents this as a proposal, not something every browser already supports.

06

Route work between the device and the cloud

Very large contexts and complex tasks may suit cloud models. A benchmark and a user-facing privacy mode can help decide when local execution is practical.

When should the application use the cloud instead? Mayes points to very large models and tasks needing a context window of around a million tokens. A larger model may solve a more complicated task more effectively than an untuned small one, though a small model adapted to a narrow set of tasks can perform similarly there.

Devices differ even within one household: Hammerly compares her new work laptop with her much older personal one. Mayes recommends a hybrid approach that first runs a quick benchmark of operations such as matrix multiplication. If the device cannot meet the speed needed for a useful experience, the application can send the work to the cloud.

Even on a capable device, the application could initially use a cloud model while the local model downloads, then switch. Mayes says the design depends on whether privacy must be maintained throughout. Hammerly suggests offering a user-visible privacy or offline mode once the model is available, with users choosing which work stays local.

07

Start with the smallest model that meets the standard

Mayes recommends testing smaller models first and adapting one that nearly works. Hammerly and Mayes stress measuring results against a defined baseline.

Mayes sees another obstacle in the relative maturity of developer tools. He says the JavaScript ecosystem has had fewer AI libraries than Python; he built his own JavaScript agent orchestration layer when he could not find one. He expects more shared frameworks and libraries to reduce that friction as the community grows.

His first advice for a new project is to define the problem and understand users and their likely requests. Start with the smallest model and move up only when it fails the task. If a model is close but misses a few edge cases, fine-tuning or distillation may close the gap without requiring a larger model that runs on fewer devices.

That rule extends beyond language models. Mayes has seen multimodal language models used for object detection when a small, purpose-built detector would do. Hammerly recalls someone using a large language model where an ordinary conditional statement would suffice. Mayes contrasts loading a model measured in gigabytes for a simple job with a much smaller model that can run in real time.

Hammerly adds that the team must define what good enough means and establish evaluation standards for the application. Mayes says to record the previous system's performance before replacing it: if a non-AI solution was 70% accurate, the new solution should improve on that measure. Tracking results gives a team evidence that the change adds value.

08

Make separate decisions about device speed and query complexity

Mayes proposes testing device performance before downloading a model. He also suggests a small routing model trained on examples to choose between local and cloud processing.

Hammerly asks how an application should decide which model to use. Mayes separates two decisions: whether the device can run a model at a useful speed, and whether a particular user request needs the cloud. For the first, he has proposed a JavaScript benchmarking system to other Google teams.

One possible benchmark would download a compact description of the model architecture without its large set of trained weights, substitute zero values and measure execution speed. If that test meets the application's threshold, it could download the actual model. Mayes presents this as an idea for a benchmarking system, not as an existing guarantee.

For routing individual requests, Mayes suggests assessing their complexity in the application's orchestration layer. A small model could be trained on examples of how people would route requests and predict whether local or cloud processing is appropriate. Hammerly notes that small models can also help identify queries that do not need a model at all.

Mayes says he previously built agent applications with a two-billion-parameter Gemma model, including a travel example and a Web AI DJ. In those applications, the small model understood enough natural language to select tools and pass them arguments. A similarly focused model, he says, could decide where to send a request.

Hammerly returns to the central design constraint: a cloud application can choose its machine, but a browser application must adapt to whichever device appears. The choice of model size, optional fine-tuning and routing between local and cloud all follow from the user's task and the device's capabilities.

For developers who want to try the approach, Mayes points to a Web AI video playlist and courses linked from the description. Hammerly closes by thanking him for explaining how to reason about on-device applications and their tradeoffs.

同じチャンネルの記事

16:07EN
Google Cloud Tech· 2026年11月21日Connecting a Personal Trainer App to Firebase with GeminiDevin Chasanoff · GoogleDevin Chasanoff walks through sign-in, database security and a live workout test with Gemini guiding each setup step.Vibe Codingコーディングエージェント開発ワークフローとテスト+2
スライド 2 枚10:21EN
Google Cloud Tech· 2026年11月19日Why AI applications use two kinds of graph—and when to skip themAnnie Wang · Google CloudA refund demo separates control of the work from retrieval of connected facts, while showing the cost of both.Agent orchestrationAI agentsWorkflow automation+1
スライド 4 枚15:19EN
Google Cloud Tech· 2026年11月17日Give a Voice Agent Memory Without Sending Its Whole TranscriptLaxmi Harikumar, Miles Libbey · Google CloudA 40-word window and persistent facts cut classifier input in a Scribant demo while keeping user preferences across sessions.Agent memoryエージェント工学AI agents+2
スライド 6 枚31:46EN
Google Cloud Tech· 2026年11月17日Build an Inspector That Makes UI States Easy to TestDevin Chasanoff · Google CloudDevin Chasanoff isolates real components, adds state presets and turns the workflow into reusable agent skills.Agent skillsコーディングエージェントAI interface design+1
スライド 2 枚16:12EN
Google Cloud Tech· 2026年11月16日Looker CLI Lets Agents Discover and Run Admin TasksChrissie Goodrich, Mike DeAngelo · Google CloudMike DeAngelo explains how command discovery and JSON templates let agents use Looker’s API without loading every tool description at once.開発ワークフローとテストエージェントMCP+2
スライド 6 枚55:11EN
Google Cloud Tech· 2026年11月14日A no-code AI app build: diagnose the system before asking for a fixDevin Chasanoff · Google CloudDevin Chasanoff builds a workout coach in stages, using tests and context tracing to fix an API connection and unintended plan changes.AI interface designSoftware engineering開発ワークフローとテスト+2