Shopify’s Mikhail Parakhin: Spend on AI to do what was once impossible

SUMMARY

In 30 seconds

Shopify gives employees unlimited tokens because CTO Mikhail Parakhin wants them to attempt work they could not do before, not just complete familiar tasks faster. He says its merchant prediction system models business actions so Shopify can explore possible interventions before offering guidance or loans. His spending rule is to use the largest models for development and research, then save on production with appropriately sized, often fine-tuned models. He also recommends safeguards against runaway processes.

1The human and the model solved a problem neither solved aloneParakhin says he found a solution to a mathematical problem he had pursued for seven years through back-and-forth with a model. He calls the ability to do previously impossible work more consequential than simply making existing work faster.
2Shopify measures completed projects as well as code changesParakhin says Shopify estimates project complexity and tracks whether teams execute more projects faster after accounting for that complexity. He presents this as a way to measure productivity gains.
3Unlimited tokens let employees test the strongest modelsParakhin says Shopify gives everyone unlimited tokens and uses the heaviest models it can find. He argues decision-makers must use the products themselves to understand what more capable models make possible.
4A simulated merchant can reveal possible paths to growthParakhin describes representing a business as a sequence of actions and testing interventions on a digital twin. Shopify can then offer a loan or suggest changes to shipping or advertising in production, he says.
5A model can warn managers before a project slipsParakhin says he changed his mind about whether models can help manage teams. He built a system to flag potential missed deadlines and signs of employee unhappiness.
6Use large models in development and save on productionParakhin recommends using the largest models for coding, testing and research, with circuit breakers for runaway use. For production, he recommends selecting and often fine-tuning a model to balance cost, throughput and quality.

Takeaways

Parakhin recommends using the largest available model for development and research, while using appropriately sized, often fine-tuned models in production.
Put circuit breakers around token usage even when employees have no ordinary token limit.
Measure engineering throughput with project complexity in mind, but also look for work that previously was not feasible.
Be precise about what is needed: Parakhin warns that models can readily build the wrong thing if asked.

TopicsAI Strategy AI cost & performance Developer productivity World models AI engineering leadership

FULL TEXT

Read the edited text by chapter

Show transcript

Transcript only

AI can raise the ceiling of what people can do

Parakhin says collaboration with a model helped him solve a mathematical problem he had pursued for seven years. He distinguishes that from using AI simply to complete existing work faster.

Mikhail Parakhin compares a company’s actions to the sequence of letters in text. Just as a language model predicts what comes next in text, he says, a company can be represented as a sequence of actions and given a digital twin.

Boris Cherny asks which difficult problems Parakhin has put to Claude and where it has surprised him. The Shopify CTO says he found a solution to a mathematical problem involving a Wasserstein loss function that he had pursued for seven years without success. He says neither he nor earlier language models had solved it.

Parakhin distinguishes this from automating work people could already do. He says the model enabled something he previously could not do, even with more time or help. For the hardest problems he sees today, he cannot solve them without the model and the model cannot solve them without him; the back-and-forth produces a combination better than either alone.

Cherny describes that as raising the ceiling as well as the floor. Parakhin agrees and says raising the ceiling matters more: ordinary work can sometimes be improved by diligence or more people, but extending what is possible has no easy substitute.

Cherny asks how to measure the business return when engineers can do something they could not before, rather than simply quantifying time saved. Parakhin says Shopify measures the easier-to-observe productivity gains carefully, including not only the number and complexity of code changes but also the complexity of projects.

Measuring productivity and giving employees model access

Shopify tracks project output with complexity in mind, Parakhin says. He argues that decision-makers should use the largest models to discover what they can enable.

Using its system for tracking work, Shopify can see whether teams complete more projects faster after accounting for complexity, Parakhin says. That allows the company to estimate productivity gains as a percentage.

Asked how a CTO should discuss the issue with a CEO, Parakhin recommends trying the largest model and comparing the results. Without that comparison, he says, a team does not know what it is missing. Shopify gives everyone unlimited tokens and runs the heaviest models it can find, he says, while investing in computing for inference, training and optimization.

Parakhin says people making decisions need to use the model and product themselves. Only then can they understand the capability, keep pushing its limits and see whether the returns justify it.

Testing merchant decisions on a digital twin

Parakhin describes a system that represents a business as a sequence of actions and explores possible interventions. He says Shopify uses the resulting information to help merchants grow.

A prediction system is one project Parakhin says he is proud of. Its premise is that a company’s activities can be represented as a sequence of actions, just as text is a sequence of letters. Examples range from opening a credit line and shipping a product to accepting credit cards or taking a loan, alongside many smaller actions by employees.

SLIDE 1 / 1A concept for testing store changes with SidekickThe video labels this interface a concept. It depicts simulated testing of proposed product-page and home-page changes before a merchant approves them.

From that sequence, he says, Shopify can create a digital twin and try interventions on it instead of on the real business. The company might explore what happens if a merchant advertises, ships a day faster or receives a loan, and look for a sequence of interactions that improves its chances of growth.

Parakhin says Shopify then acts in production: for example, offering a loan or sending a merchant advice about faster shipping or advertising. In his view, experimenting without putting real people at risk can help more merchants succeed, with a direct effect on Shopify’s financial results.

Cherny asks how long the system would have taken to build in the past. Parakhin says he does not think it was previously feasible: a company would have needed the right concentration of talent, the foresight to assemble it and the data collection to match. Raising the ceiling, to him, means doing something that previously would not even have been considered.

Using AI to spot management problems early

Parakhin built a system to identify potential schedule slips and signs of employee unhappiness. He says technical judgment matters more when models quickly build what they are asked to build.

Asked what Claude changed his mind about, Parakhin says he once thought language models could not help him manage a team. He reversed that view and built a system that analyzes ongoing activity and proactively warns him when a project may miss its deadline or people appear unhappy, so he can address issues earlier.

Parakhin says the technical side of leadership has become more important. Models are good at picking up signals and trying to please, so they can give a person what was implied by the question rather than what is actually needed. Because a model can build the requested thing so quickly, a poor initial direction may have worse consequences than before, when discussion might have intervened.

Managing engineers who supervise models is different from managing engineers who write code directly, he says. A leader has to help determine how to distribute work among AI systems, track what they are doing and avoid conflicts. Parakhin says the emphasis has shifted toward team efficiency and technical decisions rather than checking every task day to day.

Spend on development models and optimize production

Parakhin recommends broad model access with safeguards against runaway usage. He advocates the largest models for development and appropriately sized, often fine-tuned models for production.

For other technical leaders, Parakhin advises against limiting tokens too tightly, while retaining circuit breakers for runaway processes that consume resources or cause harm. He says the difference between large and small models may seem modest at first, but testing is necessary to learn what a stronger model can do.

His “barbell” strategy is to budget for the largest models in coding, development, testing and research. For production, he recommends choosing a suitable model, often fine-tuning it, and using research to balance cost, throughput and quality. Savings from production can then fund stronger models for development. He says Claude Fable 5 was a particularly large improvement in his experience.

Parakhin says he has seen teams make the opposite trade: restricting development tokens while running unnecessarily large models in production. He believes his approach will be more competitive.

Parakhin ends with a personal example: he had hoped to work in retirement on deciphering Linear A, a Bronze Age writing system used on Crete. He jokes that Cherny posted that someone had used Claude to do it before he got there.