Nebius CTO Danila Shtan on AI Clouds, Inference and Competition with Hyperscalers

Nebius CTO Danila Shtan argues that the AI-cloud market would look different if NVIDIA could produce ten times as many chips. Hyperscalers face limited supply, competing internal demand and the expense of adapting infrastructure built for other workloads.

Jacob Effron introduces Shtan and asks why the market for specialized AI clouds emerged at all. They will discuss Nebius’s origins, bringing new chips into service, inference, data-center constraints and the competition from larger cloud providers.

Effron starts with a common assumption: perhaps all an AI cloud needs is access to GPUs. Shtan, who describes himself as biased toward software, says the platform matters because customers need very different quantities of compute.

Nebius set out to support both a customer running tens of thousands of GPUs in an interconnected cluster and a self-service user who needs one GPU for half an hour. Shtan calls that ability to build at scale and sell in smaller increments a central part of the offering.

Shtan dislikes the term ‘neocloud.’ He says it once described a handful of companies but has stretched to cover hundreds. Nebius thinks of itself as an AI cloud and, in his view, its next competitors are AWS and Google Cloud rather than only other specialized providers.

He acknowledges that Nebius is much smaller than AWS, but says the company is betting on a rapidly expanding market. He expects demand to grow dramatically over the next several years, while recognizing the size of the infrastructure investment required.

Some enormous AI clusters serve their owners’ internal products rather than the broader market. Shtan also says Google, Microsoft and Meta must balance external rentals against their own substantial research and production needs.

Nebius’s bet is that an independent provider, without the same internal consumption demands, can serve customers that want access to capacity as they grow. Effron then asks how the company arrived at this position.

Shtan says he joined before the company was called Nebius. Its separation from Yandex was finalized in 2024, leaving the new company without connections to the Russian assets, he says.

Effron asks which parts of the Yandex experience transferred to a GPU cloud. Shtan points first to the infrastructure team, whose experience operating hundreds of thousands or millions of servers across facilities helped with the initial buildout and expansion.

Software was less straightforward. Nebius had to rewrite the platform from scratch and hire rapidly, so a substantial part of the software team by the end of 2024 came from outside Yandex.

The company nevertheless doubled down on a unified cloud platform rather than offering only managed bare metal or assembling a patchwork of third-party services. Shtan credits both the experienced infrastructure team and the engineers recruited during that expansion.

He contrasts the company’s emphasis with businesses that start by procuring GPUs and then finding buyers. In 2024, he says, its hardware organization numbered roughly 40 to 50 people while its software organization already had about 200.

Effron asks what happens between learning about a new NVIDIA chip and making it available to customers. Shtan says NVIDIA shares prototypes and proposed designs early, but these can change repeatedly as teams discover problems and iterate under compressed schedules.

Unlike a consumer product whose final design may be settled well before release, the AI hardware market moves so quickly that an announcement is often indicative rather than a complete final specification. NVIDIA’s decisions also affect supply chains across the market.

Once a sample resembles the production system, Nebius tests it and incorporates its capabilities and restrictions into the platform. Because it offers a cloud rather than simply bare-metal hosts, the team must ensure workload isolation, virtualization, storage and networking all work together.

At times the company has built clusters large enough to encounter software problems others had not yet seen. Engineers report issues to NVIDIA, propose or receive fixes and test them while integrating the hardware into the wider service.

New architectures change operational boundaries as well. Shtan contrasts B300-style deployments with GB300 systems, where isolation and maintenance can become rack-level questions rather than merely switching off an individual server and moving its workloads.

Effron asks how customers’ needs differ. Shtan says each customer is distinct, but their underlying workloads often fall into familiar categories: training and inference. The company initially underestimated one requirement shared by many of them: storage.

Storage is not just a question of total capacity. Throughput and latency matter: Shtan says some customers routinely pull 500 to 700 gigabytes per second from a shared volume connected to GPU servers, a scale he had not expected a few years earlier.

In his view, designing for a demanding use case—such as a multi-petabyte shared file system connected to thousands of servers—can raise the platform’s baseline enough to serve a much wider customer base.

The more important distinction, Shtan says, can be customer maturity and obligations. A newly founded AI lab may spend heavily on compute yet focus chiefly on GPU and storage performance, availability and connectivity rather than the exact location of the machines.

Enterprise customers may have smaller contracts but established clients, operating procedures and obligations across jurisdictions. They bring questions about certifications, compliance, security posture, source-code review and protection against specific attack vectors.

Effron asks whether different latency and cost requirements lead Nebius to build different clusters. Shtan says the underlying platform is intended to be general-purpose, with a high enough baseline to support a wide range of workloads.

For managed inference, customers specify traffic patterns and service-level targets such as time to first token or total request latency. Nebius then benchmarks and optimizes the service for those requirements.

Shtan says this tuning generally takes place in software higher in the stack: orchestration, model serving and the path from a customer endpoint to the inference workers. He does not normally treat it as a reason to redesign the underlying hardware.

On separating inference prefill from decode, Shtan says many people are discussing it but operating it well remains hard. Other, simpler improvements can often meet the customer’s targets first, so he does not regard disaggregation as the default for every workload.

Nebius has promising prototypes and some production deployments for especially demanding customers, but those deployments require a more hands-on approach. He says newer rack-scale systems with faster interconnects can make some earlier challenges less severe.

Which workloads fit depends on the case. A rack can handle substantial traffic, and the provider can colocate more than one managed-inference workload. The key questions are whether a workload benefits from an intentional prefill/decode imbalance and how to pack jobs efficiently.

Asked about easier inference improvements, Shtan says serving open-weight models remains far from solved. Running a newly released model with initial support in a serving engine may yield disappointing speed or cost without further work.

He identifies request processing, server configuration and kernel tuning as opportunities. In particular, optimizations can target the actual traffic pattern rather than assuming every customer has the same workload.

Shtan also sees underexplored potential in model pruning and compression, beyond techniques such as speculative decoding. He says about 95% of Nebius customers use models that have been modified or fine-tuned rather than untouched versions.

A narrower use case may need strong reasoning in one domain without preserving the same breadth everywhere. Selective compression could therefore produce a model better suited to that task, although Shtan says this approach is less suitable for open-ended research or broad coding.

For specialized enterprise agents, he expects predictability of the outcome and the real time required to achieve it to become important measures. He expects further improvements in these areas.

Effron asks which optimizations should become Nebius products and which the company can simply take from the wider ecosystem. Shtan says the company wants to invest across several layers because the gains can be additive or even multiplicative.

He describes an acquisition aimed at adding expertise in model optimization, compression and speculative decoding. Nebius combines that talent with existing specialized teams working on kernels and model compression.

Effron asks about the acquisition of web-search company Tavily. Shtan says Nebius was impressed by its product and go-to-market momentum and believed its own experience in large-scale infrastructure and web indexes could help the team move faster.

Search is not yet fully embedded across all of Nebius’s offerings, he says. The company is exploring combinations of managed inference with agent tools, including sandboxes, alongside other products still in earlier development.

On data centers, Shtan says the physical buildings may differ, but Nebius controls the internal buildout. When it rents a facility it may rent the building and electricity, while the network and hardware inside follow its own standards.

He says the same principle applies to an asset-light arrangement in which a third party supplies capital for facilities and GPUs. Standardized hardware and buildout make it easier for Nebius to operate its stack consistently and meet service-level commitments.

Effron asks which scarcity is most constraining the AI buildout: power, labor, cooling, networking, memory or GPUs. Shtan answers that the bottleneck is the ability to execute around all of them at once.

He lists shortages of GPUs, CPUs, NVMe components, facilities, electricity, qualified workers and network transceivers. Large electrical transformers can have lead times measured in years, making procurement and scheduling interdependent.

The advantage, in his view, is the ability to source components, choose appropriate trade-offs and move resources between facilities as priorities shift. He says experienced project managers are unusually valuable when so many constraints must be tracked together.

Effron asks whether changing workloads, including long-running agents and post-training, require clusters to be built differently. Shtan says a sufficiently capable underlying platform can support most of them; the larger challenge is packing and splitting workloads efficiently.

New hardware architectures can add specialized options and restrictions, but he expects fewer fundamental changes to the lowest infrastructure layer than people might assume. The platform must expose the right primitives and support colocation where full nodes are unnecessary.

Why did hyperscalers not win the market more decisively? Shtan rejects the idea that they failed to try: they already win major deals and, he says, their negotiated prices can be highly competitive even when public list prices look expensive.

One constraint is that hyperscalers must divide scarce chip allocations between internal needs and external customers. Shtan argues that a much larger supply of NVIDIA chips could have produced a very different market structure.

The business is also relatively small compared with the enormous revenue of an established hyperscaler. That affects how much attention and capital it receives, particularly when serving AI workloads requires changes to networking and storage.

Shtan expects larger clouds to invest more as the market grows and chip shortages ease. Their sales power will matter, so Nebius is betting it can grow quickly enough to compete at comparable scale when they focus more heavily on the market.

Effron asks how many specialized providers will remain in five or ten years. Shtan expects consolidation and failures, but also a long tail of independent providers, much as smaller hosting companies coexist with AWS today.

He argues that growth naturally pushes large clouds into adjacent products. That does not mean every independent supplier disappears: he points to specialist storage, compute and database companies that have found customers alongside broad cloud platforms.

Large providers will benefit from bundling and sales reach, but Shtan says a niche in a very large market can still support a substantial company. A specialized business need not be tiny merely because it does not sell every service.

Discussing OpenAI’s partnership with Baseten, Shtan speculates that offering open models can give customers a lower-cost option while retaining the ability to use frontier models. He sees the partnership as evidence of flexibility, while also noting the value of keeping users in one ecosystem.

Asked what changed his mind in the past year, Shtan points to AI coding. In his view, models and their surrounding tools have improved enough that coding assistance has moved from an unreliable novelty to a normal part of daily work.

He has personally begun using local agents and their harnesses to offload some cognitive work. Effron asks how those advances affect the management of a large engineering organization.

Shtan says he is cautious. Attractive interfaces can make it easy for low-quality material to spread through conversations and pull requests, so he wants the organization to observe how the tools are actually being used.

Nebius does not impose individual or departmental token limits, he says, but it examines usage and looks for ineffective practices. Its policy emphasizes that the engineer remains responsible for the result, including a pull request, its code and the operational consequences.

Work created by a fully autonomous agent is marked as such. Shtan says those agents will not touch core services for some time; despite working in AI infrastructure, his experience with the technology makes him conservative about critical systems.

His biggest mistake was underestimating the scale of individual customers. A 30-megawatt data center used to be a major facility; now, he says, that much capacity may not suffice for one customer.

Storage expectations have changed just as sharply. Shtan recalls when network-attached storage was measured in hundreds of megabytes or a few gigabytes per second, compared with customers now pulling hundreds of gigabytes per second from a shared system.

Asked which unresolved questions could change Nebius’s strategy, he says he assumes models will continue improving, although he is watching for genuinely new architectures. He is especially interested in optimizing agent outcomes rather than focusing only on model size.

With long-running, multi-turn agents, a customer may care less about a single request than about whether a desired outcome is reached. Shtan says a small internal research group is studying how tools and harnesses can improve the odds of success.

Some of that work brings attention back to CPU consumption, quality gates and formalized tools. An agent may spend minutes executing code and checks without issuing another model request. Shtan believes reliability and optimization outside the model will be central to wider adoption.

Effron thanks Shtan for joining the conversation. The discussion closes with the practical challenge of turning capable models and infrastructure into dependable outcomes for users.