Ahmad Osman: Local AI Needs Better Systems, Not Just Better Models

The Osmantic founder explains how hardware, tools and task-specific tests determine whether local AI is useful.

AOAhmad Osman연사 · Osmantic
채널
AI Engineer
게시일
2026년 11월 21일
영상
56:08
읽기
11분
슬라이드
7
원문
영어
YouTube에서 열기
스크롤하면 영상이 오른쪽 아래로 작아져 계속 재생됩니다. 자막은 읽기 언어를 따릅니다.
읽기 언어2개 언어로 읽을 수 있음
이 글은 아직 한국어 번역이 없어 영어 원문으로 표시합니다.
요약

30초 요약

A capable local model can fail because its setup lacks tools, software or the right hardware, says Osmantic founder Ahmad Osman. In one example, a model could configure a graphics card only after it gained search access. He recommends testing the whole workload: long prompts, generated text and parallel agents stress different resources. Smaller models can be economical for narrow tasks, but buying GPUs also brings idle time and maintenance. His longer-term bet is on systems that learn from each user’s work.

  1. 1A local answer can be fast, but one example is not a universal rankingOsman compares a local model with a cloud model on one example and cautions that benchmark results can disagree.
  2. 2Long prompts and generated answers stress different hardwareProcessing the prompt and generating tokens have different computing and memory demands, so the best hardware depends on the workload.
  3. 3Concurrent agent requests change the hardware comparisonIn the test he shows, a workstation GPU handled 50 requests in parallel while other systems were effectively sequential. He attributes part of the gap to software.
  4. 4A 100-fold component speedup can mean an 11% overall gainOsman uses a hypothetical request where tokenization takes 10% of the time to illustrate the limit of optimizing one component.
  5. 5Adding search fixed a failure blamed on the local modelThe model could configure a new graphics card after it gained access to current information and tools.
  6. 6Occasional GPU workloads may be cheaper to rentOsman says idle capacity, maintenance and reliability must be included when comparing rental with ownership.
  7. 7Osman is exploring ways for local systems to learn over timeHe proposes combining retrieval structures, decision models and weight edits; he presents it as work in progress.

핵심 정리

  1. Test models with the same tools and information before attributing a failure to model quality.
  2. Measure first response, output speed, concurrent requests and task accuracy on your own workload.
  3. Compression and inference software must suit the specific model and hardware.
  4. Compare rental costs with utilization, maintenance and reliability before buying GPUs.
  5. Use real task traces to evaluate whether a smaller specialized model is sufficient.
전문

챕터별로 다듬은 전문 읽기

원본 자막만 보기
01

Local AI needs a complete experience

Osman argues that improvements in models and hardware have to be paired with tools ordinary users can operate. Benchmark rankings alone do not answer whether a model fits a particular task.

Open-source AI is an optimization problem; local AI is a user experience problem. Both need attention at once. Since the local AI summit in June, I have seen more inference engines and more work tailored to individual models, chips and workflows.

We built an optimized inference engine for RTX Spark and used it on Windows to make a game entirely locally. Seeing ordinary visitors try it at a launch suggested to me that interest in local AI is reaching beyond specialists.

In a demonstration with Hugging Face, a local 27-billion-parameter model produced an answer comparable to a cloud model's in that particular example, at 51 versus 34 output tokens per second. One example does not settle which model is best, but it shows why local execution deserves a look.

One local and cloud response compared
슬라이드 1 / 7One local and cloud response comparedA single prompt compares output speed and first-token latency on a local workstation and cloud service.

Different benchmark rankings give strikingly different impressions of open and closed models. Some show a wide gap, while others put an open model ahead. I find it hard to know which comparison to trust, so I want to judge models on the actual work I ask of them.

Agent workloads are increasingly generating tokens for other agents. But an agent may spend much of its time waiting for a shell command or another task to finish, so raw output speed is not necessarily the limiting factor in a workflow.

My bet is on small, specialized models. A model can be strong at producing code while less reliable at unfamiliar problem solving; a specialized security model can be useful without matching a larger model on everything.

02

A trailer built across local machines

The Osmantic founder describes using an agent, GPUs and a laptop to assemble a video. He argues that different parts of a workflow can use different models and machines.

I am Ahmad Osman, founder and CEO of Osmantic. The thesis behind the company is that every step of a workflow does not require the most capable frontier model. Many individual jobs can be handled by smaller models designed for them.

I made the trailer for this talk with an agent. I gave it videos and articles, selected the material I liked, and the agent assembled clips, text and music. That work took around six hours.

The trailer still depended on a combination of machines: two RTX Pro 6000 GPUs handled tasks such as tagging and upscaling, while my MacBook handled editing and choosing between alternatives. I expect a small specialized model will eventually do such work at home.

The cloud is a machine in somebody else's data center. Portal, our agent harness, is meant to make the computing capacity at home usable as part of that workflow. The useful question is which machine and model suit the task, not whether a benchmark labels something frontier intelligence.

03

Compression and memory change what fits

Osman recounts running a large model on 14 GPUs, then explains why compression techniques and model architecture affect accuracy and memory needs.

Long inputs remain difficult because they consume memory. My older setup of 14 RTX 3090 GPUs could barely hold DeepSeek R1. Running that 671-billion-parameter model in system memory gave me roughly eight tokens per second at first, but became slow as the context grew.

Later, a smaller open model was more capable in my experience. Even earlier, I used a local model and a single GPU to build a screenshot organizer. The missing ingredient was often engineering around the model rather than another jump in model size.

Hardware-specific software is getting better, including inference engines optimized for particular GPUs. Consumer workstation GPUs and data-center chips can have different characteristics even when they share a family name, so optimizations need to reflect the actual machine.

Quantization stores model values at lower precision to reduce memory use. Different approaches retain more precision where it matters and compress other parts more aggressively. Treating every layer as identical can damage a model, so the method matters as much as the headline bit count.

Two meanings of dynamic quantization
슬라이드 2 / 7Two meanings of dynamic quantizationRuntime scaling and a file with fixed layer precision are different mechanisms.

I showed an example where a newer four-bit model retained more of its measured accuracy than an older one under compression. Better quantization can make capable models fit on less memory, although the result depends on the model and method.

04

Match hardware to the workload

Prompt processing, token generation and concurrent requests stress different resources. His hardware tests illustrate why bandwidth and software must be measured together.

A model first processes the prompt, then generates an answer. Processing a long prompt tends to depend on computing throughput, while generating each new token often depends on memory bandwidth. The shape of your workload should influence the hardware you choose.

Model architecture matters too. A dense model uses all its parameters for each token, whereas a mixture-of-experts model activates a subset. Their memory and speed demands differ even when a single model-size number makes them look comparable.

I expect much smaller models to become useful for more jobs, with another model coordinating them. Better hardware and model optimization can put capabilities that once required a data center onto personal devices, though the right comparison depends on the task.

Advertised memory bandwidth alone does not predict the outcome. In our tests, software optimization helped two smaller machines outperform a system with higher stated bandwidth. I think the software around some hardware still needs substantial work.

When someone asks which hardware to buy, I ask how much time they can spend making it work. An established software stack may be a better choice for a person who wants results now; a less mature stack can reward someone who wants to experiment.

In our Hugging Face demonstration we ran the same model on a workstation GPU, a MacBook, a DGX Spark and another system. The workstation GPU produced more tokens per second, which I associate partly with its memory bandwidth.

The gap was more pronounced when we ran 50 requests concurrently: the workstation GPU reached almost 1,300 tokens per second in the displayed test, while the other systems were effectively sequential. I see this as a software problem as well as a hardware comparison.

Fifty simultaneous requests across machines
슬라이드 3 / 7Fifty simultaneous requests across machinesThe displayed run contrasts aggregate output under concurrent load on four systems.

A general inference engine has to support many machines, which can limit how closely it is tuned to each one. I would like more engines optimized for a particular model and chip, with open-source contributors testing on the hardware they actually own.

If several agents are going to work in parallel, measure simultaneous requests rather than only the speed of one response. My preference for a GPU comes from that kind of workload; another machine might do better with an inference engine specifically built for it.

05

Measure the whole inference pipeline

Different engines and quantizations change performance. An example of a much faster tokenizer shows why a large component speedup may yield only a small end-to-end gain.

There are different model formats, compression techniques and inference engines. I do not think agreeing on one standard is the immediate priority. Experimentation can reveal which optimizations matter, even if some current implementations are discarded later.

For an inference engine, compare time to the first token, output speed per user or agent, requests per second and task accuracy on your own equipment. A number posted online may not reproduce for your combination of chip, model and workload.

The same model can perform differently on different hardware because instruction sets and inference software differ. You cannot assume that instructions or data handling will work the same way on each platform.

One team built a tokenizer about 100 times faster. Osman uses a hypothetical request where tokenization takes 10% of the time to illustrate that this would make the whole request only about 11% faster. The example shows why a component speedup is not the same as an end-to-end gain.

The effect of a faster tokenizer on a whole request
슬라이드 4 / 7The effect of a faster tokenizer on a whole requestA hypothetical request illustrates how a large component speedup produces a smaller end-to-end improvement.

People are experimenting with speculative decoding and hardware-specific versions of inference engines. A paper I had only just skimmed described scheduling changes that increased throughput in its tests; I need to examine it more closely before drawing firm conclusions.

06

A model needs the right tools around it

A local model failed to configure a graphics card until it had search and access to current information. Osman argues that user experience and model comparisons depend on the surrounding tools.

Our aim with ODS is to select an appropriate inference engine for the model and platform instead of tying the user to one engine. Most people do not want to research these internals; they want a local tool that works.

A local agent fails to configure GPU lights
슬라이드 5 / 7A local agent fails to configure GPU lightsOsman uses this example to ask whether the agent needs a bigger model or access to search.

A friend asked a local model to change the lights on a new graphics card and concluded that the model was worse when it failed. The cloud coding tool had access to search, while the local setup did not.

After I gave the local setup search access and the information it needed, it succeeded. A model trained before a tool or device appeared may need current documentation. Comparing the models without giving them comparable tools can mistake an infrastructure failure for a model failure.

A bad first experience can turn someone away from local AI entirely. That is why a complete setup, including tools and sensible defaults, matters: the user should not have to become an inference-engine expert before getting a useful result.

A paper I mentioned describes an agent constructing reusable engineering tools around repeated work. The principle is to invest in the surrounding system so that the model spends fewer tokens repeating the same steps.

07

Specialize models and weigh rental against ownership

Task traces can support a smaller model evaluated for a particular workflow. Renting can suit occasional use, while owning hardware adds maintenance and reliability costs.

People switch models at different rates depending on their work. I encourage trying newer options instead of assuming that one vendor's model will always be the best match for every task.

I cited an Airbnb example of a two-billion-parameter model achieving the quality needed for a particular task at a small fraction of a frontier model's cost. It illustrates what specialized training and evaluation can do for a defined workflow, not a guarantee for every application.

Agent traces can help create a task-specific evaluation and training data. From those examples a team can explore a small model that runs faster and costs less for its own repeated work. The result still has to be measured on that work.

Specialization also has limits. A model tuned for common coding tasks may be a good executor without being the model I would choose for genuinely novel research. Speed, cost and capability have to be weighed against the particular request.

Buying a GPU is not always the right answer. If a job runs only a few hours, renting capacity and shutting it down afterward may be cheaper than paying for equipment that sits idle.

Trust in the provider matters too. I worry that a low-cost service could route requests to a different model from the one advertised. For workloads where the distinction matters, I prefer to check the infrastructure and what is actually running.

Competition from improving open models puts pressure on pricing, in my view. The price per million tokens does not directly tell you which model will be best for your application; compare useful output and total cost for the tasks you have.

A rental service with a command-line interface can let an agent start a GPU job and stop the machine when it finishes. This still takes some engineering, but it avoids paying for an idle machine throughout the day.

Ownership brings its own maintenance: dependencies, silent failures, failed hardware and the need for redundancy. Running the equipment yourself can be enjoyable, but those operational costs belong in the comparison with a cloud provider.

08

Plan for better models and continual learning

Osman expects better engines and models to make existing hardware more useful. He outlines an experimental approach to continual learning based on lookup structures, decision models and weight edits.

Three forces are coming together: models with more capability per parameter, models specialized for a workflow, and better inference engines. A GPU bought today may run a much more capable model later because models and software keep changing.

I compared a newer 27-billion-parameter dense model with an earlier, much larger mixture-of-experts model. The newer model performed better in the comparison I showed, even though it needed much less total memory. Active parameters and total memory requirements are different quantities.

A newer model also reduced the memory used by its attention cache in the example I displayed. Techniques that make long inputs cheaper are another reason to examine the whole model and engine, not only the amount of memory on a device.

A comparison of attention-cache memory
슬라이드 6 / 7A comparison of attention-cache memoryThe slide compares reported cache sizes for different DeepSeek architectures at a fixed context length.

My next bet is continual learning: a system that adapts from the user's own experience instead of accumulating ever more instructions in separate files. I think local operation will matter because a provider serving many customers will not necessarily train a separate model for each person.

As a context grows, I have seen an agent create conflicting files or repeat mistakes despite a surrounding collection of instructions. In my view, that makes instruction files and memory workarounds an incomplete substitute for learning.

We demonstrated a local model on a Windows machine with 128 GB of memory and about 72 output tokens per second in that example. My point is to consider the model and optimizations that may be available next year when deciding what hardware to buy today.

Open models can improve quickly, and specialization is one way to make useful progress when a team does not have the computing budget of the largest labs. I had predicted a certain local capability by year-end and saw comparable performance sooner in my own tests.

An experimental recipe for continual learning
슬라이드 7 / 7An experimental recipe for continual learningOsman combines memory lookup, decision models and weight edits as a direction for further work.

I am exploring a possible recipe for continual learning combining lookup structures, decision models and edits to model weights. A lookup structure could keep retrievable information on an SSD or in memory. This is a direction I am working on, not a finished system.

Such a design might use loops within a model, around a model and across experiments. Some lookup components could run on ordinary CPUs and storage rather than requiring a GPU for every operation.

Evaluations need to reflect your own corrections and traces, not merely reward a model for exploiting a benchmark. Good data and careful tests are necessary if you want to know whether a change helps on the work you actually do.

같은 채널의 다른 글

슬라이드 3장14:28EN
AI Engineer· 2026년 11월 20일Zach Bruggeman: Ramp Made Its Coding Agent a Shared Team ToolZach Bruggeman · RampRamp made agent sessions visible to colleagues, spreading working practices and giving reviewers evidence for changes.코딩 에이전트개발 워크플로와 테스트Company Culture+2
슬라이드 6장18:30EN
AI Engineer· 2026년 11월 20일Zach Lloyd: Treat Coding Agents as a System You Can Measure and ImproveZach Lloyd · WarpWarp’s founder argues that teams can improve coding agents by recording their work, testing changes on real tasks and measuring cost and quality.코딩 에이전트에이전트 엔지니어링평가(Evals)+2
슬라이드 5장19:08EN
AI Engineer· 2026년 11월 19일Thariq Shihipar: Keep a human-readable layer in AI-built softwareThariq Shihipar · AnthropicA fighting-game experiment gives its creator readable rules to edit while an agent handles the underlying engine.코딩 에이전트Human-AI Collaboration개발 워크플로와 테스트+1
슬라이드 6장22:25EN
AI Engineer· 2026년 11월 19일Fabian Hedin: How Lovable keeps coding agents working for hoursFabian Hedin · LovableLovable separates agent history from execution, checkpoints working computers, and feeds verification results back into each task.코딩 에이전트Agent orchestrationSandboxes & isolation+2
슬라이드 5장17:50EN
AI Engineer· 2026년 11월 18일Melvin Johnson: Train coding agents to solve tasks, not game testsMelvin Johnson · Google DeepMindGoogle DeepMind’s post-training lead explains harder tasks, reviewer-style grading and rewards for honest reports.코딩 에이전트모델과 학습평가(Evals)+2
슬라이드 3장14:07EN
AI Engineer· 2026년 11월 18일Walden Yan: Build the Feedback Loops That Let Coding Agents WorkWalden Yan · CognitionCognition reports 40 times more merged changes after making its code testable by agents and assigning longer-running goals.소프트웨어 팩토리코딩 에이전트Developer productivity+2