Thomas Sohmers on AI Inference, Memory and Data-Center Growth

Thomas Sohmers describes Positron AI as a fabless semiconductor company building hardware and low-level software for generative AI inference, from chips through complete systems and rack-scale deployments. Harry Stebbings begins by asking how inference infrastructure differs from training.

Sohmers characterizes training as heavily compute-bound: the workload can process large amounts of existing training data in parallel, and more floating-point operations can help train larger models. Inference has a different pattern, particularly when a model generates output sequentially.

For autoregressive generation, the model produces a token before proceeding to the next. Sohmers says accessing model weights repeatedly makes memory bandwidth an important constraint, so adding raw compute alone does not necessarily improve inference proportionately.

The ‘memory wall’ predates AI. Sohmers says compute performance has improved faster than memory bandwidth over decades. He estimates that between 2014 and 2024 GPU floating-point performance improved about 120-fold while bandwidth improved about 17-fold; these figures are his comparison, not independently established here.

He attributes part of the divergence to differences in how memory and logic scale on silicon. SRAM cells and their surrounding circuitry have not shrunk at the same rate as some other transistor arrangements, while compute architecture also saw substantial advances.

Earlier deep-learning workloads, including convolutional networks, often benefited from more compute without the same demand for memory capacity or bandwidth, Sohmers says. He argues that the growth of transformer models made memory constraints much more consequential.

He points to GPT-3 as an important step in demonstrating the capabilities of much larger transformer models, followed by wider attention after ChatGPT. These are his selected milestones for the industry’s shift toward large language models.

Turning to token economics, Sohmers argues that cached inputs can be far cheaper to process than new computation and may produce high margins for providers. Actual cost and margin depend on the workload, pricing, utilization and infrastructure; his comparison is not a universal measurement.

He refers to a reported gross margin for Anthropic’s API business and says competition could compress margins. The interview does not establish that figure or the company’s overall profitability, and Sohmers distinguishes his enthusiasm for competition from the interests of hardware suppliers.

Sohmers disputes the idea that frontier AI providers necessarily have no path to profit. He speculates that training is a major cost and that slowing it would change their economics, but says he does not believe cost reduction is necessarily the intention behind calls to pace development.

On AI safety and pacing, he says he wants humanity to avoid catastrophic outcomes but believes the technology offers enormous potential benefits. He worries that broad calls to slow development could strengthen movements seeking to stop it altogether.

His second concern is concentration: he argues that regulation limiting advanced technology to a small number of companies or governments could create durable power imbalances. This is his political assessment of potential rules, not a finding that any particular regulation would have that effect.

Stebbings suggests companies may have strategic reasons for supporting oversight. Sohmers agrees that incentives can vary, while saying he believes many people at Anthropic take the potential benefits and risks seriously. His comments about executives’ motives are speculation.

Sohmers warns that leaders who request government intervention may not retain control over how it is implemented. He fears an eventual bureaucracy could restrict access and progress, though he offers this as a hypothetical outcome rather than a description of a specific adopted policy.

Stebbings asks what happens if the US slows while China does not. Sohmers views certain catastrophic AI scenarios as unlikely and sees geopolitical competition as a serious concern. His predictions about future Chinese policy and access are opinions, not evidence of a settled plan.

Both speakers question whether unilateral pacing would keep a technological lead. Sohmers argues that export controls and assumptions of continued dominance may not provide a durable solution; he uses military competition as an analogy for the risks of preparing for the previous technological era.

Sohmers says he generally favors free trade and exchange of ideas, while arguing for restrictions involving authoritarian governments that do not reciprocate. This reflects his policy preference and his view of China’s role in international trade.

He says public debate should consider the incentives of technology leaders and politicians. His own position is to oppose the direction of broad frontier pacing, while valuing discussion of both benefits and risks.

Sohmers alleges that opposition to data centers across the political spectrum is largely a Chinese influence operation. He does not present evidence establishing foreign direction or financing. Opposition can also arise from local concerns; Stebbings specifically raises electricity prices, water and visual impacts.

In response to aesthetic concerns, Sohmers suggests making large data centers more attractive and presenting them as engineering landmarks. He considers criticism of their appearance separate from the broader strategic value he assigns to the facilities.

He claims that some large facilities use less water than a restaurant and argues that closed-loop liquid cooling reduces consumption. He gives no supporting data for the sweeping comparison; cooling designs, indirect water use and local impacts vary substantially, so it cannot be generalized from the interview.

Sohmers connects his unsupported influence allegation to his view that China is expanding energy and data-center infrastructure while the US faces public resistance. He does not demonstrate that Chinese authorities caused the opposition, and criticism of a project is not evidence of foreign coordination.

He argues that the US has land on which to build and criticizes restrictions on some federal land, citing Nevada’s geothermal and solar resources. His assertion that remote projects would affect nobody is not established: ecological, cultural, water and community impacts require site-specific assessment.

Sohmers also claims data centers always bring enough new generation and cannot draw on power allocated to existing customers. The interview supplies no evidence for these universal claims. Grid connection, tariffs and cost allocation differ by project and can affect other users.

Asked how much planned capacity will be completed, Sohmers expects major developers largely to build their capacity, though projects may move when communities oppose a particular site. This is his forecast, and he acknowledges that local political opposition has affected projects.

Sohmers says he is open to alternative locations, including ocean-based facilities and, in the longer term, space. He names an ocean data-center company with which Positron is partnered; the feasibility and economics of those approaches remain uncertain.

On the energy constraint, he says Positron aims to increase useful compute per watt. His illustrative comparison suggests that a more efficient system could produce a given amount of work with less power; he expects customers might use the savings to do more work instead of simply building a smaller facility.

He frames energy as a long-term constraint on economic progress, but says availability alone is not enough: power and infrastructure also have to be economical. In his view, financing and the willingness to take on debt may limit the pace of construction.

Asked about rising infrastructure debt, Sohmers says he worries more about sovereign debt and inflation than about AI companies missing their targets. He expresses confidence in companies’ revenue and ability to execute, but those views are not a guarantee of financial performance.

His concern is that a sovereign-debt or currency crisis could affect the wider financial system. Stebbings then asks him to explain key-value caching and compression, topics that connect hardware constraints to the economics of serving models.

Sohmers starts with tokens: model systems divide text into reusable pieces, which can be whole words or parts of words. The sequence of input and generated tokens forms the context that the model processes.

He explains that early transformer implementations could repeat work on earlier tokens as new tokens were generated. A key-value cache stores intermediate attention keys and values so that subsequent steps can reuse them rather than recomputing all prior state.

The trade-off is storage. Cache size grows with the sequence and varies with model architecture, while the alternative requires more computation. Sohmers emphasizes that providers must decide how long to retain each user’s state and where to store it.

For compression, he first discusses quantization: representing values with fewer bits. Reducing precision from 16 bits to four can greatly reduce raw storage, but a naive conversion can damage model quality; the amount of degradation depends on the method and workload.

He describes more sophisticated schemes that share scaling information across groups of values, retaining much of the storage saving while reducing quality loss. The same general idea can apply to weights and caches, though there is still a trade-off between compression and fidelity.

Sohmers says services can run without KV caching, but he expects worse economics and performance. The value of a cache rises when sessions reuse a lot of prior context, making retrieval and retention decisions important to operators.

He cites a coding-agent benchmark as an example of a workload with extensive token reuse. The percentage he quotes describes that particular benchmark and its traced sessions; it should not be assumed for every model, customer or workload.

Large models and long individual sessions can make cache capacity significant relative to the model weights. Sohmers offers illustrative model-size and user-session numbers, some based on assumptions or reported estimates rather than disclosed specifications.

Operators must decide how much accelerator memory to devote to weights versus user state. Sohmers says persistent caches can save providers substantial recomputation and may improve the user’s perceived speed, though the balance differs with the workload.

He describes a tiered approach: keep active sessions close to the accelerator, move less recently used state to host memory, and use slower, larger flash or networked storage for older sessions. Each tier changes cost, capacity and retrieval latency.

The challenge is choosing what to retain and where as user numbers and context lengths grow. Sohmers says Positron’s product strategy reflects its expectation of larger models, more users and longer contexts; that forecast and the company’s performance claims remain to be tested in the market.

He argues that a larger context could let a coding agent work with more of a company’s repositories and supporting information. He regards access to relevant context as a practical constraint alongside, not independent of, the model’s reasoning ability.

Asked whether small enterprise-owned models conflict with his expectation of larger models, Sohmers describes two tiers. He expects frontier providers to continue building more capable large models, while enterprises also pursue smaller or local systems for cost or data-control reasons.

For Positron’s own use, he says he is willing to pay more if a frontier model produces substantially more value. Companies seeking on-premises control may choose different trade-offs because hosting the largest models themselves can be difficult.

Sohmers estimates that a small number of major providers account for most token usage and expects self-hosted enterprise models to remain a minority. He does not provide data for those market shares, and the future distribution is uncertain.

He sees small models as particularly interesting on phones and other devices. Rather than assuming every local token replaces a cloud token, he predicts local assistants could monitor relevant information and send harder tasks to larger remote models, increasing total demand.

In his hypothetical example, a local assistant watches email, a calendar and messages, then decides when a more powerful model should analyze something. The result could be more calls than a person would initiate manually, provided users trust the system and grant it the necessary access.

Why expect larger frontier models? Sohmers points to past scaling trends but explicitly says these observations are not a mathematical proof that improvement will continue indefinitely. His confidence that larger models will remain useful is a judgment based on those trends.

He compares his experience with a recent model to the excitement he felt when trying early ChatGPT, while acknowledging that future capabilities are hard to predict. He treats the model’s performance as evidence for his own optimism, not as a universally accepted definition of AGI.

Sohmers says a recent model helped with difficult coding tasks and found issues in a codebase that previous models had not solved as well. He distinguishes meaningful improvement from an assertion that all software-design problems are solved.

He also describes experiments with computer-use tools, visual work and interior design. These are his personal examples of capability; the interview does not provide an independent evaluation or establish how reliably other users would reproduce them.

For a chip-design task, he says he tested a model on an encryption block from a hardware description through a physical-design flow. He reports that the process ran in a little over 50 hours and compares it with the longer time he estimates an inexperienced human might need.

Sohmers says electronic-design tools can be difficult for models because interfaces are unintuitive and documentation may not be public. He attributes the reported success to a combination of computer use and scripting. His results are anecdotal and should not be treated as independently verified benchmarks.

Stebbings asks how to interpret AI labs developing their own chips. Sohmers welcomes competition and says different engineering teams can pursue distinct architectures. As a hardware-company executive, he expects varied approaches to expand the available design space.

He says Positron has made its own architectural choices while the rest of the industry is also evolving. In his view, customers and market outcomes will determine which approaches create value; he does not claim that any design has already won.

On context length, Sohmers says conventional attention and hardware impose practical limits. He claims Positron’s next generation will have substantially more memory capacity than a leading competitor’s device; that is a company projection, not a verified comparison of shipping products.

He says expanding context from a million tokens to much more can be difficult under existing memory and computation constraints. Sparse, linear and other attention mechanisms may reduce those costs, but their usefulness depends on implementation and quality.

Sohmers credits Chinese model developers with important algorithmic work under hardware constraints, including multi-head latent attention. He says compressing attention state can exchange additional computation for a smaller cache, illustrating that improvements can come from algorithms as well as chips.

Asked whether algorithmic savings reduce the need for new hardware, he says there may be quality or other trade-offs. He speculates about adoption by US labs based on his information and rumors; the interview does not establish those companies’ undisclosed architectures.

A larger advertised context window is not the same as effective use of every token in it. Sohmers says some earlier models lost recall as context grew, and cites long-context tests as one way to examine whether relevant information can actually be recovered.

He cites his own comparison of recent model results on a retrieval-style benchmark. Those figures are his reported observations rather than a controlled independent comparison, and one benchmark does not measure every long-context task.

Stebbings asks about a reported decline in token prices from around $60 per million to less than $1 over several years. Sohmers says price alone misses an important difference: a token from a more capable model may be useful for tasks that an earlier model could not perform well.

He compares older and newer model capabilities and offers a very large estimate of the increase in value per token. That is his qualitative economic judgment, not a standardized measurement; realized value depends on the application, errors and alternatives.

Would pricing move from tokens to useful outcomes? Sohmers thinks per-token pricing will remain common because it is relatively easy to calculate and bill. Outcome pricing is harder to define and verify, although he can imagine other plans for more autonomous products.

As a hypothetical, he describes a future provider offering a fixed annual price for extensive use of an agent capable of employee-like work. The number and capability in his example are speculative, not an announced product or price.

Stebbings asks what might burst an AI bubble, including the possibility that frontier providers miss growth expectations or users shift toward open models. Sohmers agrees that such a slowdown is possible but says he considers it less likely.

His reason is the same complementarity thesis: an inexpensive local assistant could perform routine monitoring and ask a more capable cloud model for help on harder tasks. That might increase cloud demand, although it relies on user trust, access permissions and the economics of those calls.

The conversation turns to companies supplying data and services for model development. Sohmers sees potential for growth but says a key uncertainty is whether frontier labs will eventually integrate more of those tasks themselves rather than buying them externally.

To close, he emphasizes what he calls human alignment: getting societies to coordinate on desirable outcomes as AI develops. He says technical safety work matters but may be constrained by disagreements over regulation, energy, data-center locations and other social choices.

Sohmers believes economic incentives shape decisions and hopes more people will analyze these coordination problems carefully. Stebbings closes by noting the breadth of the discussion, from infrastructure and model economics to policy and the future of AI.