Scaling Slack and GitHub: Maude Lemaire on Load, Shards and Reliable Platforms

Gergely Orosz introduces principal engineer Maude Lemaire, author of Refactoring at Scale. They discuss scaling Slack and GitHub, load testing, database migrations and how she uses AI while investigating performance at Cursor.

Lemaire grew up around computers: her father taught computer science. She originally planned to study genetics, however, and began university in microbiology rather than assuming she would enter software.

She realized that being good at memorization was different from enjoying it. While considering a switch to engineering, she took an introductory computer science course and discovered that she enjoyed making a computer do things.

Her first job was at Rent the Runway, combining her interests in programming and fashion. She admired engineering leader Camille Fournier, who later helped her find an opportunity at Slack.

Orosz recalls evaluating messaging systems for a large company around 2016. The discussion sets the context: a product that worked well for smaller teams could encounter different limits when one customer needed tens of thousands of people connected.

At Slack, Lemaire’s onboarding colleague introduced her to performance work. She says many early improvements involved revisiting calculated shortcuts that helped the startup move quickly but became expensive as usage grew.

Slack was preparing its enterprise product while IBM was an early large customer. As adoption expanded, the company created a dedicated enterprise performance team; Lemaire joined it and learned alongside more senior engineers.

She describes the architecture of that period, not necessarily Slack today. Data was sharded by team or workspace. A large customer could receive a larger dedicated database, while smaller customers shared an instance, but eventually the largest available machine became a limit.

WebSocket connections were also associated with a customer’s server. Busy organizations could be assigned dedicated chat infrastructure, an approach that worked until enterprise-wide usage challenged the assumption that a large company could simply be split into smaller workspaces.

Slack added an enterprise entity above workspaces so people at the same company could collaborate across teams. That created a hot point: enterprise-wide direct messages and large shared channels made many users query the same enterprise shard.

Even populating a user’s sidebar involved multiple data representations: public channels, private channels, direct messages and group messages had been stored separately. Joining those tables became costly when customers grew.

A colleague working on notification counts concluded that incremental tuning was no longer enough and the data needed to be unified. Lemaire volunteered to undertake the migration while the old tables were still serving production traffic.

The PHP codebase contained SQL directly in many procedural call sites, rather than behind a conventional ORM. She found more than 400 relevant calls and needed a way to change them without silently changing their behavior.

She first moved the existing queries into clearly named functions, preserving what they did instead of optimizing them at the same time. This minimal data-access layer made subsequent migration work more manageable, while feature flags controlled individual changes.

The migration used dual writes and backfills to populate the new table, followed by shadow reads that compared new results with old ones while production still used the old path. Dashboards tracked discrepancies before any switch.

During an internal rollout, she found a discrepancy and reran a familiar delete-and-backfill procedure, forgetting that colleagues were already reading the new table. Slack employees briefly saw empty sidebars; she switched reads back to the old data while the backfill completed.

For load testing, Lemaire starts with the product’s actual usage patterns and likely failure modes. At Slack, one was a thundering herd: many users reacting to the same event at nearly the same time.

Her example is a CEO posting good news in a channel with 100,000 active users. Many recipients open the channel, fetch its history and add reactions within seconds; each reaction can itself fan out to many connected clients.

Slack prioritized event types under heavy load: text messages mattered more than reactions, so some reaction events could be dropped. Lemaire calls this a product judgment as well as an engineering decision.

She also questions whether a message needs unlimited additional reactions. She says Slack limited distinct emoji types but still allowed additional votes on those types; deciding what users can lose requires a product owner and context.

An event flood is different from a thundering herd: one client is overwhelmed by incoming events. The two can overlap, but the unit of failure in an event flood is the receiving client.

She recalls an IBM administrator signed into many workspaces who could receive the same enterprise message repeatedly because connections were not deduplicated. His client could become unusable, even though he was one of the product’s most committed users.

To prepare for a customer expecting roughly 500,000 users, Slack needed a test that included live WebSocket connections. API-only load tests could exercise database requests but missed client event delivery and downstream reactions.

An earlier approach ran Slack in headless Chrome instances across Kubernetes pods. Lemaire says it was high fidelity but expensive and operationally fragile: the team spent substantial time bringing crashed pods back rather than running the desired scenarios.

She quotes roughly 37 cents per headless instance for a five-hour test, and the hosts estimate a very large total cost at the target scale. The discussion is an approximate historical estimate, not a current cloud price.

The replacement used lightweight Go processes that maintained WebSocket connections and issued representative API requests. It sacrificed some browser fidelity but exercised the path the team cared about at a more practical resource cost.

The simulation also included requests to edge caches used for latency-sensitive data. Lemaire stresses that a useful test needs to model the real path, including both event delivery and the actions triggered after an event arrives.

The team evolved the tool into routines for simulated users. Lemaire built a recorder that captured API calls and incoming and outgoing WebSocket messages during a real product flow, then turned selected events into a repeatable script.

She wanted product managers to record important workflows themselves and send them to the performance team for regular testing. Product specialists know how a feature is actually used and can identify cases platform engineers may overlook.

Lemaire joined GitHub in 2024 as the company was handling growing load and moving more infrastructure toward Azure. A colleague told her that physical capacity was constrained, so the challenge was both to migrate and to use existing resources more efficiently.

She had worked on moving Slack from team-based sharding toward a more dynamic arrangement using Vitess over MySQL. A channel could be a more useful partitioning unit than an entire large team, though a sound key depends on access patterns.

At GitHub she found, at that time, 16 shards for pull requests and issues. She expected many more and asked why the number was so small. The answer involved scatter queries that had to touch multiple shards.

If a query lacks the sharding information needed to route it, it may have to search every shard. She gives finding a user’s pull requests as a hypothetical illustration, explicitly saying she does not know whether that exact query scattered.

More shards could improve distribution, but a scatter query would then touch more places. Lemaire initially wanted to remove scatter queries; she found that doing so in a large Rails monolith was more difficult than applying the same playbook elsewhere.

Active Record made the underlying database behavior less visible than the explicit SQL she was used to. She describes pairing with an experienced Rails engineer to trace a small function whose generated query was difficult to locate.

Her concern is that abstraction can make it easy to add code without seeing the database work it causes. She also recognizes that the monolith had supported substantial growth; the difficulty is changing long-established access patterns safely.

An internal GitHub debugging view showed the queries and clusters used to render a page. Lemaire recalls a repository page requiring over 100 MySQL queries across at least three clusters when she last checked; this is her historical observation, not a current measurement.

She questions whether every visible count needs to be calculated live, especially once a value is large. Product requirements affect caching choices, so the appropriate tradeoff depends on which numbers users actually need to see precisely.

Asked about AI, Lemaire says it has helped with implementation more than with the investigative part of performance optimization. At Cursor she investigated time to first visible token, which depends on work before and after the request reaches a model provider.

Preparing context, cache decisions and the path through an inference proxy all affect latency. Some data cannot or should not simply be cached indefinitely because of product, privacy, load and cost constraints.

She finds language models useful for basic instrumentation but less reliable at choosing what to instrument in an asynchronous flow. She also uses mutation testing to challenge generated tests that merely confirm the implementation they were written alongside.

Cursor had both too many and too few useful traces, she says: traces with millions of spans were hard to use, while sampling in inconsistent places could remove needed information. She manually examined the flow and instrumentation to understand synchronous and asynchronous work.

She had a model generate a large sequence diagram from prompt submission to the first visible token, printed it and annotated it while reading the code. The diagram was a starting aid; she checked dependencies and correctness herself.

One proposed optimization was to parallelize tasks. Lemaire says that was not necessarily useful when very few users needed some of those tasks and one long task still dominated latency. In her account, simpler changes such as extending an appropriate cache TTL could be more useful.

She uses an LLM throughout the day for exploration and routine engineering, then verifies important results manually. Her concern is that a platform foundation needs to cover the whole problem, not merely the easiest portion of it.

For engineers interested in platform work, she recommends moving between product and platform teams. Repeatedly seeing the same problem in product work can reveal an opportunity for a shared solution, while rotations help platform engineers stay close to real usage.

She argues that stronger platform guardrails are especially valuable when product teams can ship much faster with AI. Orosz summarizes the reciprocal lesson: understand customer needs, performance and reliability, and collaborate across the boundary rather than treating the teams as separate worlds.