Forward Future Live: GPT-5, AI Infrastructure, Evaluations and Coding Agents

Matthew Berman opens the August 15, 2025 Forward Future Live with co-host Hiten Shah. They preview GPT-5's mixed reception, semiconductor export rules and AI economics, and introduce guests Jordan Nanos, Tejal Patwardhan and Guy Gur-Ari.

The hosts introduce the episode's three guests and describe the week's mixed GPT-5 reception. The program begins with news before the interviews.

Berman enjoyed GPT-5 in his tests, but viewers had very different reactions. Shah felt the new model had lost personality and says the rollout showed how much people valued GPT-4o and GPT-4.1. His impression of API routing is a user observation, not a confirmed description of every API request.

Shah describes the launch as a rapid release into an uncertain new field. He calls it a 'YOLO' model, a characterization of the rollout rather than a claim about the team's actual process.

Berman suggests OpenAI underestimated users' attachment to a conversational style. Models and system instructions change over time, but an entirely new model can make the difference especially noticeable. He notes that GPT-4o was brought back after the reaction.

Berman suggests changing a model's weights or system instructions can affect its style. The large shift in user experience after a model replacement makes those changes visible to people who relied on a familiar tone.

Shah thinks a chat interface encourages people to relate to a model as they would to another person. Berman distinguishes emotional attachment from a practical problem: existing workflows and prompts may have been designed around a particular model and cannot always be swapped unchanged.

Berman mentions a prompt conversion guide for users moving from GPT-4o to GPT-5. Shah says he had already used structured prompts with earlier models and sees the transition as part of learning how to communicate requirements.

Shah says his input-and-output-oriented prompting already resembled some of the advice being offered for GPT-5. He sees users learning how to get consistent answers and perceives fewer hallucinations. Berman reports a similar personal impression, not evidence that hallucinations have disappeared.

The hosts ask how to measure the less tangible qualities of a model, such as tone and personality, alongside math and coding scores. Shah describes using human judgments to develop graders and then applying a model to score outputs, while remembering that real users have varied goals.

Shah suggests establishing human-scored examples and using them to develop automated evaluation. Berman contrasts measurable technical tasks with conversational qualities and plans to ask Patwardhan how her team approaches them.

Berman raises reports that NVIDIA and AMD agreed to share 15% of revenue on certain China chip sales in exchange for export licenses, followed by reports of Chinese concerns about those chips. Shah interprets the arrangement as a negotiation; the hosts treat China's motives as an open geopolitical question.

Shah points to China's investment in its own semiconductor ecosystem and says the chip arrangements involve multiple parties. The participants acknowledge that outside observers do not know every negotiating objective.

Jordan Nanos joins from SemiAnalysis, where he says he works on ClusterMAX, testing neocloud GPU providers. He describes the firm's semiconductor research, newsletter, consulting and data products for industry participants and investors.

Nanos argues China supports domestic accelerators and manufacturing through investment, subsidies and policy. He says shifting developers to alternative chips could slow immediate work while building capability over time. He links a reported DeepSeek delay to this transition, but acknowledges the account relies on reporting and inference, not confirmed causation.

Nanos suggests adapting to unfamiliar accelerators requires engineering effort and a shift in priorities. Such work could delay near-term model development while potentially building an ecosystem over the longer term.

Nanos cites reporting on DeepSeek and notes its earlier release cadence. He doubts a delayed model will necessarily meet expectations, but does not claim direct knowledge of the company's internal schedule.

Asked about timing, Nanos speculates that production at useful scale could take five to ten years, with regulations, equipment access and continued investment major uncertainties. He distinguishes reaching useful scale from overtaking TSMC in advanced manufacturing, which he expects would take longer.

Nanos discusses Huawei Ascend versions and supply chains. His claim that the chips discussed are all made on TSMC processes should not be treated as established: manufacturing and packaging can differ by version and export restrictions. The exact composition and availability of specific products require independent verification.

Nanos expects the AI-chip market to grow without a single winner taking everything. The hosts then discuss a meme about AI applications earning $100 million in recurring revenue while facing a $120 million model-provider bill; the numbers are illustrative, not evidence of any particular company's accounts.

The hosts' meme uses hypothetical negative margins to question the economics of applications built on external model APIs. Nanos distinguishes the sustainability of a specific pricing scheme from the usefulness of the products themselves.

Nanos says coding assistants have become useful enough in his own work that he doubts users will willingly give them up, even if economics must change. The hosts compare current venture subsidies to early discounted Uber rides and past high SaaS valuations.

Berman recalls subsidized rides in the venture-capital era. Shah compares the moment to periods when ordinary software companies received extremely high revenue multiples and says investment still requires real execution and demand.

Shah describes the race as funding growth and technology until demand and execution support a sustainable business. Nanos points to earlier venture-backed companies that lost money before reaching scale, and expects the market to find a price matching model quality and speed to what developers will pay.

Nanos uses ordering a car at an airport as an example of a convenience users may not want to give up. He expects providers eventually to settle on combinations of price, speed and model quality that customers value.

Nanos divides GPU buyers into three groups: managed-cluster customers running training or inference, wholesale buyers managing their own infrastructure, and on-demand users who need to spin capacity up and down. He places AWS, Azure, Google Cloud and Oracle among major providers, with CoreWeave a large AI-focused player.

SemiAnalysis tracks more than 130 neocloud providers, Nanos says, ranging from converted crypto miners to sovereign and venture-backed clusters. They largely buy accelerators, install them in data centers and rent capacity. Gigawatt-scale projects represent very large announced investments, not proof that every proposed buildout is complete.

The buildout includes repurposed mining facilities and long-term commitments as well as hourly rentals. The hosts note that operators weigh AI infrastructure against other possible uses for power and equipment.

Nanos attributes CoreWeave's position partly to a user experience built around AI clusters, support and software. He describes neocloud prices and service in 2025 as competitive with hyperscalers for some startups, while cautioning that larger cloud providers are also competing aggressively.

He frames the future as a contest between differentiated features that justify a premium and the possibility that customers need only a limited set of cluster capabilities. The hosts turn to custom silicon, including Google TPUs and AWS Trainium.

Nanos ranks accelerator ecosystems from his user perspective, putting NVIDIA first, followed by Google's TPU, AWS Trainium, AMD and Huawei, with other specialists in the mix. This is an analyst's judgment for 2025, not an objective ranking for every workload.

Custom silicon also includes CPUs, networking and storage, Nanos notes, not only AI accelerators. For a customer choosing infrastructure, the practical question is how the entire system supports training or inference.

He discusses Google's TPU partnerships and speculates that broader external availability could become a larger competitive issue for NVIDIA. Customers must weigh total cost, performance and researcher productivity, not simply chip specifications. Claims about individual contracts, ownership stakes and restrictions are his contemporary understanding, not independently established here.

Berman asks why Google would expand TPU access. Nanos sees an opportunity for revenue and partnerships, while Shah notes that some external access already exists through cloud arrangements and asks whether Google has enough capacity for itself and customers.

Shah observes that Anthropic and other labs can source compute from multiple providers. He distinguishes selling TPU access through partnerships from making it broadly available and asks whether capacity would permit that expansion.

Nanos refers to colleagues tracking accelerator shipments and says they see production ramping, while noting the limits of his own visibility into proprietary data. He believes demand substantially exceeds supply, based on what he sees from model developers and their rollout constraints; this is an assessment rather than a published industry-wide measurement.

Nanos says model providers have reported difficulty serving their latest systems to all free users. He sees routing as one way to match scarce resources to queries, although the supply-demand picture is based on incomplete outside information.

Nanos says matching the right model to each query could help allocate scarce compute. He also speculates that intent-based advertising or referral links might fund inexpensive requests. Berman thanks him and moves to the next news story.

The hosts discuss a reported unsolicited $34.5 billion Perplexity offer for Chrome. Shah suspects a publicity strategy; Berman asks whether owning the browser would be strategically valuable. Their account of motive is speculation, and an offer is not an acquisition.

Shah sees AI browsers such as Comet and Dia as applications that use websites as context and automate browsing. He thinks the browser may become commoditized or unbundled into specialized experiences. Berman sees the browser as an important surface for AI interaction.

Berman says a browser can be a strategic entry point for AI and describes his own easy migration to Perplexity's Comet. Shah suggests that future desktop products could have browser capabilities specialized for a particular use.

Asked which platform matters most, Shah chooses the operating system and argues that Apple should not be counted out: its devices, chips and software integration could support local models. Berman agrees Apple has resources but says his own experience with Apple Intelligence has been disappointing.

Shah says Apple's ability to integrate its operating system and chips could matter for on-device AI. Berman notes that Apple has followed earlier innovators successfully before, though he sees its current AI launch as a different challenge.

Shah argues Apple may need to ship and iterate rather than wait for a mature category. He praises its writing tools, while Berman describes moving to a Pixel to try Google's AI features. These are individual product impressions, not a systematic comparison.

Tejal Patwardhan joins from OpenAI, where she says her evaluation work measures model knowledge, skills, tool use and behavior quantitatively. These results inform research, safety and launch decisions. Berman asks how evaluation should evolve as familiar benchmarks saturate.

Berman asks how evaluators move beyond a saturated academic test. Patwardhan views benchmark gains as real progress but says the next challenge is measuring complex, open-ended tasks that better resemble work.

Patwardhan says teams increasingly want complex, open-ended, longer-horizon tasks resembling real work rather than only exams with self-contained answers. Berman raises SWELancer, based on software freelance tasks, and PaperBench, which tests research reproduction.

She says SWELancer's dollar framing makes the scale of completed work more interpretable, while PaperBench tests a common research activity: reimplementing prior papers. Replication could be a building block for research assistance, not proof that an agent can autonomously improve itself.

Patwardhan says reimplementing existing research helps new researchers learn methods and find issues. A model that assists with that activity could support research, but it would be only an early step toward more independent discovery.

Berman asks whether the freelance benchmark provides a blueprint for earning money. Patwardhan thinks a model may help a freelancer work faster but leaves the practical question open. Shah asks how evaluators choose use cases.

Patwardhan describes two goals: deliberately cover important areas such as coding and science, while testing a broad range to assess generalization. She cites math and programming competition work as an example of evaluating skills across domains, without claiming that one result solves general intelligence.

On personality and tone, Patwardhan says she works mainly on frontier capabilities but describes systematic prompts tied to the Model Spec, repeated sampling and grading, A/B tests and researcher impressions. No one metric fully captures the range of user preferences.

Asked about her own informal tests, she says she sometimes asks for a joke but spends more time on evaluation dashboards, identifying failed tasks and skill gaps. Her priority is generalization across many examples rather than one favored prompt.

Berman asks whether PaperBench points toward self-improving AI. Patwardhan says reproducing research is one early building block; understanding shortcomings and proposing new ideas would be further steps. She finds both the potential and the need to measure it important.

The question of self-improvement calls for careful measurement as well as excitement, Patwardhan says. In her framing, a model must first reproduce work before it could be expected to diagnose and extend research reliably.

She says a surprise in PaperBench was how willing paper authors were to help create detailed rubrics, partly because models might reduce tedious research work. She also observes that once a benchmark becomes a target, teams often make progress on it; improved scores still need interpretation.

Berman turns to emergent misalignment. Patwardhan describes work in which problematic training can produce undesirable behavior beyond the narrow examples used. The [OpenAI study](https://openai.com/index/emergent-misalignment/) examined specific models and synthetic training settings; it does not show that all misalignment is easily detected or fixed.

Patwardhan says researchers identified an internal feature associated with a misaligned persona and explored detecting it early. They also found that relatively small amounts of additional training could reduce the measured behavior in their experiments. These are bounded experimental results, not a universal safety guarantee.

Berman asks how interpretability can find concerning behavior when internal computation is not fully understood. Patwardhan distinguishes specific, tractable investigations of latent features and chain-of-thought monitoring from the much harder general problem of understanding a model.

Patwardhan identifies focused interpretability questions, such as tracking a particular latent feature, as a way to make progress without claiming a complete theory of model computation.

Berman presses on whether chain-of-thought text faithfully reveals internal reasoning. Patwardhan says optimizing too directly against visible reasoning may encourage models that conceal problematic reasoning; she discusses reducing such pressure while acknowledging faithfulness cannot be guaranteed.

Asked whether everything a model can do can be benchmarked, Patwardhan says evaluation is difficult. Creative writing illustrates the issue: people can disagree about quality. Shah judges writing partly by its intended audience; Berman says he finds it harder to distinguish good from great.

Patwardhan contrasts objective problems with verifiable answers and subjective tasks requiring human or personalized judgment. Randomized comparisons and human preferences can still quantify aspects of subjective behavior, but the result depends on who is asked and how the experiment is designed.

Human comparisons can assign people to different models or let the same person choose between outputs, Patwardhan explains. Such experiments can make subjective preferences measurable without pretending that every user wants the same answer.

The discussion turns to personalization. Patwardhan gives emojis as an example: some users like them while she prefers a concise answer. Berman recounts telling ChatGPT to avoid fluff only to receive repeated assurances that there is no fluff; Shah prefers a somewhat conversational and proactive style.

Shah describes tuning a system prompt to make the assistant conversational and proactive, and hesitating to change a configuration that works for him. Berman connects this preference to the broader discussion of model personality.

On safety, Patwardhan describes a layered approach: filtering some pretraining data, evaluating behavior, safety training including safe-completion work, and deployment monitoring and enforcement. Multiple layers can cover different gaps, but no layer guarantees perfect safety.

Shah asks how the GPT-5 router was benchmarked. Patwardhan says she did not work closely on it; her team compared the underlying models with predecessors. She emphasizes controlling for scaffolding, prompts, test-time compute, verbosity and sampling when interpreting headline accuracy.

Even a single model can produce different answers on repeated samples, Patwardhan notes. Evaluation comparisons need to account for settings and compute rather than treating a headline score as an isolated property of model weights.

Patwardhan is excited by seeing model capabilities develop and wants more work on alignment monitoring and interpretability. She also expects wider usage to improve understanding of model behavior and personality. The hosts thank her for joining.

After the interview, Shah says headline benchmarks may matter less to users than evaluations for specific jobs, such as accounting and finance. Berman describes asking GPT-5 to build a budget and forecasting app from business financials, which he had not yet reviewed.

Shah says models sometimes make mistakes when transforming financial data but can surface patterns worth investigating. Their examples are personal experience; financial outputs still require checking. Berman wonders why ordinary accounting software cannot provide natural-language reporting more easily.

The hosts discuss Google's Gemma 3 270M open-weight model and local inference. Shah wants evaluations on the actual target device and use case, including comparisons with API workloads, and notes progress in local text-to-speech.

Shah says a useful local-model test should run on the actual computer or phone intended for deployment. He wants to compare that performance with the same tasks currently sent to a remote API.

Berman contrasts a future of large general models with more specialized models and edge compute. Shah imagines browser-like apps processing web context locally. Both see opportunity, but describe it as underexplored rather than demonstrated at scale.

Berman wants local tools that make web search and synthesis as easy as hosted chat. Shah expects some workloads to move from paid cloud inference to devices; Berman adds potential privacy and latency benefits alongside the continued need for large cloud models.

Guy Gur-Ari joins to discuss Augment Code, which builds coding assistance for teams with large codebases. He describes completion, next-edit, chat and an agent. He later corrects Berman's introduction: he is co-founder and chief scientist, while Matt McLaren is CEO.

Gur-Ari says the company's core thesis is that stronger models still need relevant context. His team spent roughly a year exploring ways to gather context from large codebases, carrying the approach from completions into agents. His claim of competitive differentiation is a company representative's assessment.

He recounts his physics background and work at Google on understanding machine-learning models and solving math and science problems. He joined systems engineer Igor Ostrovsky to build coding agents for large enterprises, incubated at Sutter Hill Ventures.

Gur-Ari says his Google research explored the internals of machine-learning systems and difficult reasoning tasks. Coding offered a chance to apply related ideas to a product for large organizations.

Asked about pricing, Gur-Ari says Augment then charged by user message. As models make more tool calls and accomplish more from one instruction, users may gain value but the provider's inference costs can grow without the per-message price changing. The pricing described is historical.

Per-message pricing has a tension: a more capable agent can do more on a single instruction, which benefits the user, while its extra model and tool calls raise the provider's cost. Gur-Ari says the team was considering how pricing might evolve.

He explains the alternative tradeoff: charging for every token or tool call can penalize a user when an agent loops or fails. A message is the instruction the user sends; subsequent model calls, tool calls and tests were not counted as separate user messages under the plan he describes.

Shown the $100 million revenue versus $120 million model-bill meme, Gur-Ari acknowledges expensive inference and corrects his title. He expects competition, cheaper models and possibly open weights to reduce costs, but says companies must balance optimization with a stable user experience.

He predicts a shift from one agent doing everything toward multiple agents that can use different models. Berman asks whether the product could route tasks automatically so users need not pick models. Gur-Ari calls this speculative but plausible.

Switching models midway through a session loses the benefit of the existing model's cached context: the new model must process it again. Gur-Ari says a quality improvement must justify that cost, and sees this as a limitation on mid-session routing in a simple single-agent design.

Gur-Ari says routing is easier at the start of a session than after a long context has accumulated. Multi-agent designs could offer more flexibility in deciding what context each model needs.

His preferred multi-agent pattern has an orchestrator handling the high-level plan and user interaction, while subagents perform bounded tasks. That can keep detailed context out of the supervisor and provide one point of contact. He notes decentralized agent swarms as a more complex alternative.

The hosts ask whether subagents would use the same model or different ones. Gur-Ari says it remains open and task-dependent: a system might mix hosted models or open-weight models, creating flexibility but not an automatic guarantee of lower cost or higher quality.

On edge inference, he is cautious. Frontier models then exceeded typical laptop capacity, though smaller subagents could potentially run locally. Supporting varied user hardware adds product complexity, so he says any cost, speed or quality benefit must be meaningful.

Gur-Ari describes early GPT-5 observations in Augment: strong tool use and thorough codebase exploration, with readable structured answers, but sometimes more tool calls before completion. He stresses that prompts and the surrounding product shape behavior, and that this is anecdotal early feedback.

He says model behavior in Augment is shaped by prompting and integration, not just the base model. The team was still improving the GPT-5 integration, so his initial observations could change.

Asked about a one-million-token Claude context window, he says Augment manages context dynamically so sessions and codebases can extend beyond a single model window. A larger window can still help by holding more relevant material at once, but costs may rise and his team was still testing it.

Gur-Ari calls the product's experience effectively unlimited in context because it retrieves and manages relevant information, not because the model has an infinite context window or perfect recall. Berman and Shah note the distinction and ask about his preferred coding model.

Gur-Ari says Claude Sonnet 4 had been his daily driver until recently, after earlier versions 3.5 and 3.7, but declines to name the newer model he is trying. Berman jokes about GPT-6; Gur-Ari denies it. The hosts later speculate without evidence about the unnamed model.

He says moving between Claude versions required adjustments, and understands why some users preferred the behavior they already knew. He does not disclose the newer model he is testing.

Berman thanks Shah and the guests and closes the livestream. The recording ends with the hosts' farewell; the interviews contain their questions and answers throughout rather than a separate audience Q&A section.