Two Weeks, Thirty Compiles: How Adam Majmudar Wrote a GPU From Scratch

A person who had never designed a GPU wrote one in two weeks. TinyGPU can run matrix addition and matrix multiplication, the code is open source, and the full open-source chip flow takes more than twenty minutes and about fifty steps on his laptop. He kept thirty-plus failing runs before the design hardened without errors. The college path from computer architecture to actually designing a chip is usually two or three years. Majmudar’s bet was that you can cut that time by about 100× if you are willing to trade encyclopedic thoroughness for the slice of the field that actually matters.

Nathan Labenz invited the TinyGPU author onto Cognitive Revolution to take the GPU apart from software down to hardware. The feeling of the conversation is double: the core logic of a GPU is simple enough to sketch in one episode, and AI is quietly changing how fast a single person can learn a closed industry. Labenz also frames the project as a monument to a multigenerational stack. Majmudar could only do this work because thousands of engineers spent decades on chip design and design-automation software.

Majmudar had spent three years on a company he started in college, then stopped and began a series of industry deep dives, sharing the work on Twitter. He picks technologies he thinks will matter over the next one to two decades, and areas he has cared about for years. Each time, the question is not only “can I do the technical work?” but whether he can also read the industry: tailwinds, who is winning, and where a young person might actually have a shot.

Chips were the first dive, planned as two two-week sprints. The first sprint was the whole engineering chain: EDA, the software that turns an architecture into a layout; fabrication; and architecture itself. His rule is closer to 99/1 than 80/20: 1% of the concepts carry most of the value. Three years of school buys coverage and detail; two weeks is for pulling out that 1%. The second sprint needed a project that would test the learning. GPUs sit under graphics, AI accelerators, and tensor machines, so they made a natural foundation. TinyGPU came out of that second sprint.

Almost nobody does this as a hobby, and Majmudar starts the explanation by contrasting GPUs with CPUs. CPUs are old enough that a college architecture course will teach control signals and how machine language becomes action on a chip. Software and hardware are connected, so a student can replicate a CPU from scratch. GPUs are the opposite. NVIDIA and AMD publish high-level architecture diagrams and almost nothing that dumbs the machine down into pieces you could implement. Intel has released some low-level diagrams, but those are production designs, too dense to use as a textbook.

The tools are a harder wall. Production EDA from Synopsys and Cadence costs hundreds of thousands to millions of dollars per seat; an individual cannot touch them, and shipping a production layout still requires that moat. What made TinyGPU possible is a stack that appeared over the last decade. OpenROAD, a DARPA-funded open-source EDA project, is not feature-complete next to the commercial tools, but it is enough for an individual or a small project. SkyWater open-sourced a 130-nanometer process design kit, a real foundry flow even if the transistors are large. Efabless lets a group share a shuttle: about $10,000 for 100 chips. Tiny Tapeout, created by Matt Venn, carves one of those chips into roughly 100 slots so a crowd can split the cost. Majmudar calls Venn the person who made ASICs accessible, and points to Venn’s Zero to ASIC course.

On top of OpenROAD sits OpenLane, the flow aimed at the SkyWater stack. Majmudar plans to submit TinyGPU to Tiny Tapeout. He will get a chip that holds his design plus about ninety-nine other people’s projects, and he can play with all of them. Labenz asks what the stack is good for besides learning. Majmudar does not expect huge businesses on hobbyist tools; the real production stacks will keep winning. He does see a possible market for small-ASIC prototyping, and a clearer need: get students an intuition for chips while the CHIPS Act and geopolitics are starving the industry of people.

Looking back, Majmudar boiled the project down to three words: architecture, parallelization, and memory. The first two are obvious from the outside. Memory is the one he completely underestimated. He also had to decide what “GPU” even meant. The word now swallows graphics processors, tensor machines, and assorted accelerators. His product goal was teaching, so he stripped graphics-specific hardware and kept the skeleton shared by all of them. If someone says it is not a GPU because it cannot draw triangles, he does not care.

The software pattern is SIMD, single instruction, multiple data; the hardware name is SIMT, single instruction, multiple thread. Take two 1×8 matrices, A and B, and write the sum into C. A CPU uses a for-loop and spends eight iterations. A GPU kernel only knows how to add one pair of elements: load A[i], load B[i], add, store C[i]. You then tell the machine to run that kernel eight times, once per thread. The trick is that eight copies of the same code still need different addresses. TinyGPU gives each thread sixteen registers; the last three are read-only, and the GPU writes them: which block you are in, how many threads per block, and which thread you are inside the block. From those values the kernel computes a memory offset. Anyone who has written CUDA will recognize `blockIdx`, `blockDim`, and `threadIdx`. TinyGPU runs threads in groups of four, one block per core. Block 3 holds global threads 12 through 15.

The instruction set is a teaching constraint, not a product ISA. Instructions are 16 bits with a 4-bit opcode, so there is room for at most sixteen ops; register fields are 4 bits, so there are at most sixteen registers; branch immediates are 8 bits, so program memory is at most 256 lines. Real GPU instructions are usually 32 bits and much looser. The ops themselves are plain: add, subtract, multiply, divide, load, store, constant load, plus a no-op and a return. The pair that makes kernels interesting is CMP and BR_NZP. CMP subtracts two registers and stores whether the result is negative, zero, or positive in an NZP flag. BR_NZP jumps if that flag matches a condition. Matrix multiplication needs a dot product and a loop; those two instructions are how a thread accumulates.

Then memory, which Majmudar calls his biggest correction. The naive story of parallelism is compute: a hundred additions need a hundred adders. The thing that actually stalls is bandwidth. A hundred units trying to read a hundred global addresses will queue, and latency blows up. He uses a sharper picture: a thousand cores asking at once, bandwidth for a hundred, everyone else waiting. He happened to show the design to George Hotz, who told him it had not solved the memory hierarchy — “the most important thing.” Hotz’s line, as Majmudar quotes it, is that a GPU is largely the task of working around memory latency and hiding it at different layers. Real GPUs grow memory controllers, caches, shared memory, and barriers for that reason. TinyGPU keeps only registers and global memory, and it cuts barriers. The same ceiling shows up in today’s models: training and inference sit on GPUs, and vendors keep stacking cache and bandwidth to hide the wait.

Once the software model is clear, the hardware is easier. Everything starts on the host. The CPU loads a compiled kernel into program memory, loads the matrices into data memory, writes a thread count into a device-control register, and raises a start signal. The GPU runs, raises done, and the host reads the result. GPUs are not meant to live alone; they always sit next to a machine that compiles, loads, and collects. Majmudar left out the real host-to-device bus to keep the teaching design small.

The first block inside the GPU is the dispatcher. It watches four cores and uses the simplest policy that works: round-robin. A core exposes done, reset, and start. When a core finishes, the dispatcher resets it and sends the next block of four threads, like a cafeteria line that feeds whichever window is free. In code it keeps two counters, blocks dispatched and blocks done; when they match, it raises done to the rest of the GPU. Labenz asks whether he dragged the dispatcher off a menu of parts. Almost nothing in the project was off the shelf. Majmudar designed each block from how he thought it should work, then used Claude to check the idea.

Each core shares a scheduler, a fetcher, and a decoder. The fetcher reads instructions from program memory and caches them, because every thread in the block is running the same kernel. The decoder turns 16 bits of ones and zeros into control signals for the per-thread execution units. Each thread then looks like a miniature CPU: its own register file, program counter, ALU, and load-store unit. Real GPUs must handle branch divergence, when threads jump to different lines. TinyGPU gives each thread a program counter, but Majmudar wrote kernels that keep every thread on the same instruction and sidestepped the hard case.

A neat hardware detail: the adder, multiplier, and the rest of the ALU are wired up all the time, so they compute every cycle. A multiplexer chooses which result to keep. “Executing an instruction” is the decoder opening the right switches. Memory controllers do the throttling. Four cores times four load-store units is sixteen requestors that may fire together; the controllers queue those requests, release them at the memory’s bandwidth, and route answers back. Labenz asks why four threads issued together do not finish together. Without memory, everything is deterministic and every instruction costs a fixed number of cycles. DRAM is asynchronous, and a bandwidth queue means some threads get data first. In Majmudar’s design the early ones wait. Even whole cores can finish early if their data came back first. The clock is the other physical constraint: flip-flops capture a new value on the rising edge, a cycle is a few nanoseconds, and the period must be longer than the longest path a signal has to travel. Static timing analysis is the check that those paths fit.

Majmudar wrote the functional logic in SystemVerilog. After that, open-source EDA takes over. Synthesis turns the code into logic and simplifies it toward an equivalent smaller form. That logic is mapped onto standard cells for a specific process, then placed and routed: tens of thousands of gates laid down and wired. The flow repeats optimizations, checks timing, and deals with parasitics — wires that sit too close pick up capacitance, and electrical effects like that can kill a chip. The output is a GDSII file for the foundry, plus a layout-versus-schematic check that the final polygons still match the original design.

Hardware parts company with software here. Software can ship and then change. A chip that is wrong is wrong, and a tape-out starts at tens of thousands of dollars, so verification carries a weight software people rarely feel. On an M2 laptop the OpenLane flow takes twenty to twenty-five minutes and about fifty steps. Majmudar kept thirty to thirty-five local runs before the first one finished without errors. The messages are poorly documented and often random; debugging transparency is low, and you mostly have to push through. When it finally worked and he opened the layout visualization, he says the moment felt magical. Labenz’s line about standing on the shoulders of giants is not decoration: that decades-old stack is what a two-week project is actually using.

For functional checks he wrote a Python script that pretends to be the external memory, because DRAM is not part of the GPU design. The test kernels are machine code he compiled by hand. The script loads program and data, sets the thread count, raises start, and steps cycle by cycle. The log shows each thread’s registers, ALU output, and core state; matrix addition takes on the order of 500 cycles, and then it prints data memory so you can see the correct sums. He tells curious listeners to clone the repo and read that log. If he tapes the design out, the physical chip will talk to external DRAM in the same way the simulator does.

The part of the project worth stealing is the learning method. GPU internals are proprietary; there is no textbook on how to write a dispatcher. Majmudar leaned on two open GPU repos, including MIAOW and a Verilog GPU project, but they are thinly documented, production-shaped, and too big to read. His move was to see a module name, talk to Claude from first principles about where that block must sit and how it must work, state a hypothesis, let the model push back, and only then check the repo. After you have invented a version, other people’s code becomes readable. He compares it to aliens finding a computer: they will not understand it by staring at transistors. They have to form their own picture and test it against the machine they have. That, he admits, is an over-complex analogy for walking a GitHub repository.

He also noticed that Claude and ChatGPT sometimes show an engineering intuition that goes past what is written down. Details that were never published — how softmax is done on a proprietary GPU, for example — still get a plausible direction out of the model. Labenz recognizes the feeling from cramming the overlap of AI and biology, and from planning a macOS app. Static pages hand you answers. An AI that will argue with your hypothesis produces a different depth of understanding. At the end of the episode Majmudar puts it more sharply: his learning philosophy is aggressive, he thinks people can move 100× or 200× faster than they expect, with less complete depth but as much or more practical knowledge, and this project would not have been possible without AI.

On the industry, Labenz wonders whether getting this far in two weeks means too much value sits on the design side while manufacturing is underpriced. Majmudar says he is not experienced enough for a strong take. His suggestion is to read Hamilton Helmer’s *7 Powers*, which explains why moats make sense and why a two-week speedrun does not threaten incumbents. He cites a then-viral tweet from someone who had just left NVIDIA: now that he was out, he could confirm that nobody would catch the company in the next ten years. Majmudar thinks claims like that are probably right. The industry is so central, and the incumbents so competent — NVIDIA and TSMC among them — that they do not get to become lazy.

The challenger he wants to highlight is Atomic Semi. A founder who once built a semiconductor fab in a garage is trying to make cheap, relatively fast fabs, first for a mid-market that looks a bit like FPGAs, at costs on the order of hundreds of thousands of dollars instead of billions. If it works, the bet is a new market: keep a fab next to you, iterate without a $10 billion foundry queue, and serve custom designs that do not need the leading edge. As for how deep is deep enough, Majmudar is clear. He wants a panorama plus the key intuitions, enough to go toe-to-toe with working engineers and understand every word. He does not want to sink years into one corner unless a real frontier problem requires it.

The conversation ends where it started: TinyGPU is a working artifact, not a stack of notes. Majmudar already has people filing issues and contributing, more than he expected, and he treats that as part of the learning. A repo that runs will show you the holes a private notebook will hide. Labenz comes away with a stronger intuition for what a GPU is doing under the models we train, and with a picture of Majmudar as someone who will keep speedrunning whatever field he picks next.

The portable lessons are blunt. Write the 99/1 list for a new domain until you can name the load-bearing concepts; for a GPU they were SIMD, memory hierarchy, and scheduling. When a mechanism is opaque, guess a version first, then check it against a model or against source. End the sprint with something other people can reproduce. And when a program is slow, look at data movement before you look at the arithmetic — the same conclusion GPU architects keep rediscovering.