Frontier vision models score 12.5%: Joseph Nelson on the real waterline of computer vision

Nathan Labenz, host of *The Cognitive Revolution* and founder of Waymark, welcomes Joseph Nelson, CEO of Roboflow. The platform supports more than a million engineers and more than half of the Fortune 100 as they turn proprietary image and video into a competitive advantage. Labenz’s own detour into CLIP, BLIP, and BLIP-2 in 2022–2023, trying to make sense of small-business photo libraries, is the excuse to catch up on a few years of progress in about 90 minutes. Nelson starts from research history. Vision was home for early deep learning — ImageNet, MNIST, cat-versus-dog. Then language jumped ahead with Attention Is All You Need in 2017, five years of scaling laws, Chinchilla, and GPT-2, and a breakout with GPT-3, GPT-4, and ChatGPT in 2022. That five-year delay from transformer paper to nearly a billion weekly users is now happening in vision, because the vision transformer arrived in 2020.

Modalities are crashing together; brains get more context from language and vision at once. The differences still matter. Nelson’s biological analogy is a big LLM-style reasoning engine in the head plus rods, cones, and a visual cortex that decides in a fast, lizard-brain way. Biology dedicated more neurons to vision than to any other sense, and he expects machine-learning systems to specialize the same way. In practice that means a lot of vision runs at the edge, at low latency, in the real world: a manufacturing line, a product being shipped, cells under a microscope, a telescope hunting galaxies, a robot. Language jobs can often assume a long-running data-center job. Visual jobs often cannot.

Language only exists where people do, and in systems humans have crafted. "The world's much bigger than just language." Anecdotally, the number of distinct scenes in a day is more diverse than the unique words you read, so the long tails are fatter. About a million developers download Roboflow open source every 30 days; about half the Fortune 100 build on the platform. The production sample runs from a Catan player who wants a camera to prove the dice are cheating him, to a Canadian YouTuber, Dave’s Armory, who built a flamethrowing weed-killing robot and a self-driving couch, to instant replay at Wimbledon and quality assurance at Rivian. Nelson’s belief, which he thinks the rest of the world is coming around to, is that visual AI will be bigger and more important than language alone, because AI reaches its potential out in the world. He is optimistic we are approaching the ChatGPT moment for vision, with infrastructure coming online and a Cambrian explosion of places where consumer products will disappoint if they cannot see.

Labenz’s working definition of a solved problem is: is there any vision task we could not solve if we really put our minds to it, even if cost and latency are ugly? Nelson’s definition is stricter and closer to the ChatGPT aha — you ask the model and it delights you, the way you no longer train a sentiment classifier, you just talk. By that bar a subset of vision is solved, but it is not nearly as solved as language. First principles: Unicode encodes all text in memory more efficiently than a single image of three RGB channels, 0 to 255, pixel by pixel. Walk through a day writing down every object you see and you get a bell curve: person, car, food in the fat middle; changing the oil under the hood on the long tail. Because those tails are fatter, it is taking longer to represent them in models.

Some tasks feel quote-unquote solved. Count people in this image. OCR, increasingly: a GLM model he is excited about can run in real time, answer how much the salsa cost on a receipt, or pull a house address from Google Street View, almost always correctly. Multimodal models — Gemini, open models like Molmo, the DETR family of transformers — are pushing outward on that bell curve so more of a given day is understood zero- or few-shot. If you live in the middle of the curve, it is a solved problem. If you ask for something surprising that was never represented in training, you find the gap. Expectation is getting faster either way.

The other complexity is the edge. You want answers now, on a webcam or a phone, and you do not have the benefit of waiting 40 seconds. Heuristically Nelson sees about an 18-month delay between a SOTA capability in a multimodal cloud model and something that runs on a Jetson Orin-class computer or an iPhone. That delay, plus the fat tail, is why vision still feels unsolved even when most problems are tractable if you are willing to fine-tune and pay any inference cost.

Labenz has not recently had an image use case a frontier model simply could not see, though he remembers ARC-AGI puzzles where models failed because they could not even describe which boxes were which colors. Nelson’s team spends time setting expectations: zero-shot, few-shot, or more representation required. They round up vibes-based evals from Twitter onto visioncheckup.com — take your LLM to the optometrist. Grounding, meaning segmentation and detection, is a frequent gap. He is surprisingly disappointed when a model cannot place a word in a crossword; treat the same clue as pure text and it does better. Measurement and other inherently precise, pixel-perfect jobs are in the same bucket. Hearsay from the labs is that post-training cares more about user intent, with segmentation as a tool call, so pixel-level accuracy is under-solved.

Speed is still a disappointment on the Pareto. Using Gemini 3 to auto-label, he waited 40 seconds per image, and generative non-determinism made downstream results messy when he needed precise, consistent labels. Roboflow maintains playground.roboflow.com to compare SAM 3, Gemini, and Claude Opus; he finds failure cases, reports them, and they do not reproduce — not because of Roboflow’s wrapping, because the model itself does not. After speed, reproducibility. After that, representation: a lot of the world is still not understood, including where one thing sits with respect to another.

Few-shot helps, but is not infallible. At NeurIPS they published RF100-VL: a basket of 100 open datasets from Roboflow users, spanning industrial, healthcare, flora and fauna, documents, and miscellaneous. Models were asked to do successful segmentation from the same instructions a human annotator would get. Evaluated: Gemini, SAM 3, OpenAI models, OWL-ViT 2, Grounding DINO. The best at publication was Gemini 2, at 12.5% across domains. This was not a perfectly curated research set like COCO or Objects365. These are the places people actually use models. A CVPR competition on a 20-dataset subset, RF20, asked how much 1–5 image examples lift performance. The maximum lift for a single model was around 10 percentage points — meaningful from a 12% base, not a panacea. Nelson is still bullish on few-shot image-text prompts. The story is clear: we need better representation of real-world problems, and we still have a ways to go before “totally solved.”

Reasons to walk down the Pareto from a frontier API: faster response, lower cost, edge deployment, not sending data over the wire, plus owning your AI as IP, privacy, and security. Navigation starts from the job to be done. Many vision problems need an instantaneous response — something going down a line, a live sports broadcast — which puts you in a real-time class, colocated compute, and a model whose weights you own. That is why Roboflow invests in publishing models and supporting open-source repositories. Nelson is optimistic about open-source AI both as a way for everyone to realize the benefits and as a bottoms-up discovery engine.

The rising pattern is distillation. SAM 3 is a promptable open-vocabulary segmentation model: find all the sheep, people, hockey players. It is state of the art, not infallible. A customer that does instant replay clipping at Wimbledon and the US Open brings compute to the courts because a live broadcast has sub-10 milliseconds to put something over the wire. There is compute that could run SAM 3; it was not economic. They do not need an open vocabulary that includes poker chips and deer. The move: take prior-year video, prompt SAM 3 for people, tennis ball, court, and net, auto-label a high-quality specific dataset, then train a smaller model they own — RF-DETR, then SOTA for real-time segmentation and detection. It is efficient enough to run multiple streams on a single A100. Cheaper inference opens more possibilities. Use a model to failure; if it works, make it yours and fit it to the edge.

Even as models eat more of the task, there is no shame in a harness. You can ask Gemini to count tennis players and get a number; a detection transformer returns person boxes, and a tiny bit of fast code counts the class outputs. Traditional post-processing is still allowed. Streaming video to the cloud nonstop is expensive across many streams versus owned compute. Language has flipped from one-model-to-rule-them-all back toward domain-specific experts; vision was already in that camp because you are often compute-constrained where the system has to run.

Labenz wants scaling-law intuition: if zero-shot fails, is it 10 in-context examples, 100 for a small fine-tune, 1,000 next? And how does that relate to model size when the task is narrow? Data volume is informed by how many nines you need before production. If the alternative is no eyes on the thing — staffing a retail store or a museum where people do not check out — an 80% counting model might be ship-it. Healthcare manufacturers of IV bags are the opposite. An escape defect, particulate matter in the bag, is life-threatening. They need very high recall, and they are comfortable adding vision to augment people, lab inspections, or sampling, then weaning toward more reliable systems. Accuracy tolerance is a business question.

How many images depends on how varied the scene is. On one end, self-driving in the open world. Nelson recalls Andrej Karpathy’s CVPR talk about five years earlier: find a stop sign, it is the same red octagon, right? Wrong. Blocked by a bush, only-if-you-are-turning-right, on a school bus, temporary, on a gate you can see only when it is up. Petabytes of video are not sufficient for a car. Contrast a manufacturing line making EV batteries. Defects vary in length and presentation, so OpenCV rules are brittle, but the cross-section scan of a battery is far more finite than the open road. In a controlled environment you can probably get away with hundreds of images and still produce something of utility.

Model-size intuition holds: smaller is faster and usually lower recall or precision. The RF-DETR family runs nano through 2XL. At 2XL, a fine-tune is more accurate than a fine-tuned SAM 3 and 40 times faster — a fixed class list, not open vocabulary. At the small end, pico or nano models do 180-plus frames per second on a Jetson Nano with 4 GB of RAM. Oranges on a grocery rack versus particulate in an IV bag is the difference between a small model that clears the business floor and a larger one. Getting away with a smaller model is the wrong framing; deploying at higher scale with less compute may be the most optimal strategy. Harder problem, more data, bigger model: the expectations hold.

Labenz asks who publishes the open-source vision models, whether Chinese labs are distilling a shortcut, and whether an empty American bench appears if Meta’s new AI leadership loses interest. Nelson thinks there are reasons to worry about the future of open-source model supply, and reasons for optimism. In visual AI the US has almost never led, whereas in language it has been ahead on closed and open models alike. Manufacturing’s weight in the Chinese economy, and vision’s weight in manufacturing, tell you why seeing is a high priority. Names: Alibaba’s Qwen team, with Qwen3-VL world-class and competitive with closed models on vision-language reasoning, though recent researcher changes might make that tenuous. The GLM team, a 9-billion-parameter mixture-of-experts model surpassing closed models on OCR. DeepSeek’s OCR paper, whose real innovation was a data-processing trick: feed a screenshot of a page to get more tokens than readable text, because the loss in understanding was much less than the compression gained.

The US is not empty. Meta publishes the Segment Anything family; SAM 3 is the best open-vocabulary model globally. Back through FAIR and Yann LeCun: Detectron 2, Faster R-CNN, DETR, the DINO family. People dunk on Meta’s language models and under-credit how consistently good it has been at visual AI — which makes sense for a company that has to understand photos on social networks and is betting on glasses. Microsoft has the multimodal Phi family. The Allen Institute has Molmo, though funding turbulence is topical. On diffusion: Mistral, and Black Forest Labs out of Europe. NVIDIA has put a ton into open-source; Nelson thinks they now have the most open-source model repositories by count. Nemotron, Cosmos Reasoning, and a conversation with Nader, a director of open source, about making those models increasingly multimodal.

Where does a company that is not a Manhattan-sized data center fit? RF-DETR retook SOTA for the US on real-time object detection and instance segmentation. Before that, LW-DETR and related DINO-family work from Chinese labs were ahead and tougher to fine-tune. RF-DETR is the first real-time instance-segmentation transformer, and the fastest and most accurate they claim for pixel-wise segmentation and detection. The bet, which Nelson was not sure would work: a narrow task, a small model useful on the edge, under-addressed. They took a DINOv2 backbone, shared-weights neural architecture search, an Objects365 pretrain that transfers to COCO and user fine-tunes, plus segmentation and detection heads. Roboflow has raised about $63 million across rounds — not nothing, not foundation-model billions. Detection model last April, segmentation in the fall, Apache 2.0, unlike YOLO variants that now need a commercial license. “Your tiny friends at Roboflow are putting the US on the map.” The LW-DETR team in China has responded but not beaten the work back. If Meta or NVIDIA stopped publishing tomorrow, all of open source would take a hit, because improvements smash the best ideas together. He is optimistic NVIDIA would fill a Meta-shaped hole.

Labenz hears neural architecture search as a dabble in recursive self-improvement and wants the Eureka, not just the grind. Ordinary NAS is informed grid search: train a model, measure speed and accuracy, repeat. Weight-sharing NAS, detailed in a paper they published, trains thousands of subnetwork configurations in parallel in a single run. At each step one subnet is sampled by randomly choosing patch size, number of decoders, number of queries, input resolution, attention windowing — they use deformable attention. At inference you can sample any of those subnets. A NAS run is not as efficient as a single training run, but it is not 7,000 times worse. That was a huge unlock for a $63 million company trying to release models. They also had to rewrite deformable attention for inference engines that did not support it, waiting on TensorRT in the NVIDIA ecosystem until it did. DINOv3 is out; one can imagine the experiments.

Anyone can now run NAS on their own dataset. It outputs a Pareto of speed versus accuracy; you pick where to sit. When it worked on Objects365 they asked whether it worked downstream, and they rolled out hosted cloud GPUs that do NAS on any dataset. If you NAS on your data, literally no other architecture lands on those optimizations — a one-of-one model. For humans there is an easy button: run NAS, spin up subnets, one click on the curve. The product thesis is interoperability and progressive disclosure of complexity with strong defaults. Someone could reimplement NAS; making it easier engenders trust that benchmarks reproduce. The open-source `inference` package is `pip install` away, with vision-specific GPU optimizations (resize on CPU, and so on).

Agents are becoming the biggest user. Roboflow exposes CLIs so Claude Code, Codex, and the rest can take platform actions — “go optimize a model for me.” MCP versus CLI is an ongoing debate; at minimum the CLIs exist. They are taking inspiration from Vercel, which layered a chat agent on top of frontend and backend cloud so it chooses infrastructure while it builds. Roboflow’s not-yet-released Workflows AI agent would take “count people crossing the line” or “watch cars in the intersection.” The hardest part is discerning intent. Once you know Labenz meant cars at a particular crossing, SAM 3 versus a pretrained RF-DETR that already knows cars is the easy question, and you can ask which of the intersection’s lines he meant. Ease of use with good defaults, agents using the same CLIs, and a first-party sherpa: that is the product stack. You should not need to know every subnet of a training run to get value.

Labenz’s company Waymark makes 30-second TV spots for small businesses and is being asked to help with display ads. The old pain was aesthetic evaluation. Open-source aesthetic models, and a company he thinks was called Every Pixel, could tell the top of a distribution from the bottom; the middle was mush. Foundation models now advise which images would make a business proud, but they are slow and expensive to grind through a huge library. Is there a small open-source model for this, or is it still too esoteric?

Aesthetics is tough because the tasks models can recursively improve are the ones you can benchmark. The second you can benchmark, you scale compute and Sutton’s bitter lesson takes hold. Taste is in the eye of the beholder. People prefer Midjourney’s look to ChatGPT’s to Gemini’s. The LAION aesthetics predictor, released because that team also did generative images, is still the best purpose-built thing Labenz has seen. Facebook will not let text cover more than X percent of a display ad — a rules-based constraint, not a taste constraint. Nelson’s suggested approach is RLHF: a vibe-check model tuned on a given client’s brand guidelines and history of ads, perhaps few-shot. The catch is that marketing’s top-order task is to stand out. Adhere to the guidelines and you may be stylistically correct and fail to cut through noise. Short answer: no great zero-shot aesthetics model beyond what Waymark is already doing. Longer answer: this is the distinction between tasks you can post-train to victory with objective metrics and loosey-goosey ones that live outside toss-compute-at-it.

Horizons Nelson is scanning split into continuation and new S-curves. Continuation: transformer everything. Diffusion transformers and vision transformers keep eating tasks; RF-DETR is that recipe applied to real time. More nascent: self-supervision in the DINO family. DINOv3 showed you can get good latent understanding as a backbone without large amounts of supervised labels, then attach a detection, segmentation, or captioning head. The unsupervised unlock, unlike next-token prediction or CLIP’s captioned images, is billion-scale images and a student-teacher recipe — patches, gram anchoring — so a bigger teacher validates a student. If you have seen enough bedrooms, you know a lamp usually sits on a nightstand. It still feels a little unintuitive: treat patches somewhat independently, like bag-of-words for images, and you get scene understanding. Weak spatial reasoning is the cost; cross-attention is how later techniques put things with respect to one another. They did not release the dataset; he thinks it was a billion-plus images.

New S-curves. World models — V-JEPA, World Labs, various techniques — aim to understand scenes and places with a new architecture, multimodal by default, sometimes next-scene video prediction, sometimes diffusion of a viewpoint. Will they give true physics, spatial reasoning, and open-ended tasks? Probably; the interesting question is on what time horizon, and there he is not sure. Practical today: Cosmos Reasoning as a world-model-ish system for synthetic data at minimum, navigating a space at maximum. He thinks of vision as read access, robotics as write access, world models as a blend; you need read to write. Sora 2, he argues, has some of those underpinnings. Second curve: vision-language-action models. Instruction in, robot acts — move the computer 50 centimeters left. NVIDIA’s GR00T, Google’s RT-2, plus younger startups. Expect the same spread of model sizes and generality, and expect them to be edge-ready, because embedded intelligence has to run in real time. He thinks VLAs are still under-indexed.

Third, not unique to vision: inference-time scaling. Vision becomes a tool call in a broader agentic system. Nelson uses Gemini constantly as a replacement for instruction manuals — what does this remote button do, how do I relight the water-heater pilot (a high-stakes task for a farmer’s son). Search the model, pull the instructions, proceed with caution. That category, a big compute budget and post-training plus test-time compute, is just getting going, and it gives rise to visual agents: organize my images, sort brand-aligned display ads. Coding agents showed what happens when you let something run unencumbered for a long duration; vision agents will get similar benefits, with the usual speed and latency caveats. The pendulum, he says, is swinging back to vision: physical AI, multimodality, hardware, what is defensible when SaaS can be rewritten by code generation. “Water’s warm, been here all along.”

Wearables pull the constraints together: cannot be too heavy or hot, must understand the scene or they are more annoying than valuable. Roboflow partnered with Meta on the SAM family and broader visual understanding, with day-one support when SAM models launch. For Christmas his partner bought him Oakley Metas — dogfood the work. First pair that were his; he had tinkered with Spectacles and the Vision Pro. He is pleasantly surprised. Wearables are going to inflect. Eight million pairs sold last year versus 60 million AirPods: already real volume. The Oakleys target activity. He cycles and runs; they do bone conduction for music and understand the scene, though onboard AI is not there — the phone does the heavy lifting, “Hey Meta” for feedback. On a Sunday ride friends did not know they were glasses that play music and capture media; they tried them and wanted a pair for the next ride. That was the arrived feeling. This is the worst it will ever be.

He has Charlie-Brown-and-the-football timing on AR. Roboflow started as AR apps in 2017, for fun, before there was a company; he thought they had arrived, and he was wrong. They came back in 2019. The unlock is form factor: not a glass brick in the hand. Snap’s Spectacles were the first publicly traded company to mention Roboflow in an earnings statement; they integrated so developers could create custom lenses. Someone counted stop signs on a walk to prove the neighborhood was safe. Hardware has gotten good enough, and consumer willingness shows up in the numbers. He hopes the ecosystem stays open, or becomes more open, so anyone can publish apps. His bet, with no inside information, is closed APIs now to curate first-run quality, then an app-store or even Android-like sideload later. Hardware-platform adoption precedes software adoption, and we are just now starting the hardware S-curve.

Asked to paint how vision changes daily life in the next few years, he walks a day. Food produced with fewer pesticides because you sprayed only where you saw weeds. Eggs visually assured through the supply chain. A washer that can finally tell whites from colors. A fridge that restocks eggs via Instacart. A self-driving commute with zero accidents because cars talk at the intersection, and the workday starts on Wi-Fi in the car. Pixel-perfect remote colleagues in the next chair. Thursday Night Football with real-time stats and a vision agent running your fantasy team. Same-day packages checked through factory vision, delivered by a bot, watched by a Ring camera. A toothbrush that does cavity scans. None of that is theory. Every link is something Roboflow customers are working on. He jokes they will be able to power Santa Claus. Society will judge always-on cameras the way it judged smartphone cameras: quality-of-life gain versus new behavior. He takes the bet that simple wins — turn-by-turn in cycling glasses — pull people in. Privacy rights should be strongly enforced and updated, the way search-and-seizure was written before cars; public versus private space needs means-tested law on new devices. The front-page bad-actor story will come. The vast majority of what he shows is quality of life: cancer research, ocean cleanup, EVs made correctly, food with fewer pesticides.

Labenz’s last question is whether a new right might be to be classified by the smallest, most narrow model for the task — a violence detector that only sounds an alarm, not a general reasoner that can identify you by gait. Nelson spent time in Washington as a Senate intern. His general stance is the importance of open source for freedom to tinker, and that AI is a tool by which societal outcomes are realized or not. We already regulate fraud and violence. A scammer who uses an LLM to impersonate someone should be prosecuted for the scam, not for the size of the model. Minimal-model-size rules run into trouble because capabilities advance quickly and distillation is common, so you catch accidental corner cases and stymie adoption that was well-intended.

Steel-man a rule that AI touching patient health needs a governance body. Sounds reasonable. Then consider a UNC Chapel Hill postdoc using Roboflow to count neutrophils under the microscope — hundreds of colonies, a menial, hand-cramping task that tells you whether an experiment worked. That person would never endeavor to do it if “patient health” required approval. You wanted to protect patients and accidentally froze a lab. The medical system already holds people accountable for practicing correctly. Succinctly, he would not think a narrow model size nails the way this technology should unfold. Regulate misuse of behavior; regulating at the tool level is rife for accidental slowdown of what he views as a modern industrial revolution whose quality-of-life gains we will not fully forecast. Let it flourish, and stamp out bad action. The conversation, he notes at the close, could be entirely different in six months.