Most people use ChatGPT and Claude every single day and have no idea how they were actually built.
Save this :)
A small group of people understand the exact pipeline that turns raw internet text into a model that can write, reason, and code. And understanding that pipeline changes how you use these tools forever, because you finally see what's happening under the hood instead of treating it as magic.
The difference between those two groups is not a math degree.
It is one clear mental model.
Here is the truth almost nobody explains simply: every frontier model, GPT, Claude, all of them, is built through the same five-stage pipeline. The companies differ in scale, data, and a thousand engineering details, but the shape of the process is the same everywhere. Learn the shape and you understand how all of them are made.
Let me set expectations honestly before we start. You are not going to train a model that rivals GPT or Claude from scratch on your laptop. Those models cost tens of millions of dollars in compute and require enormous engineering teams. That's not the goal here. The goal is to understand the pipeline so deeply that you could build a tiny working version yourself, reason about how the big ones behave, and stop being mystified by any of it. That understanding is worth far more than most people realize, and it's completely within reach.
Here are the five stages, in order, exactly as they happen.
## Stage 1: Data — The Foundation Everything Is Built On
Before there is a model, there is text. An enormous amount of it.
The first stage is gathering and preparing the data the model will learn from. For a frontier model this means a staggering quantity of text: a large fraction of the public internet, books, code repositories, and more. But raw text is messy, so most of the work in this stage is not collecting, it's cleaning.
The data gets filtered to remove junk, duplicated content is stripped out (the same paragraph appearing a thousand times would warp what the model learns), and low-quality or harmful material is filtered down. This cleaning matters more than people think. The old principle holds: garbage in, garbage out. A model trained on cleaner, higher-quality data learns better than one trained on more but messier data. Data quality is one of the most important and least glamorous levers in the entire field.
Then comes the step that surprises beginners: tokenization. The model can't read text directly. The text gets broken into tokens, which are chunks roughly the size of a word-part. The phrase "tokenization" might become three or four tokens. Every piece of training data gets converted into these tokens, and from that point on the model only ever sees numbers representing tokens, never letters. This is why models sometimes miscount the letters in a word: they never saw the letters, only the token.
The output of this stage is a massive, clean, tokenized dataset. Nothing has been learned yet. You've just prepared the raw material.
What to Do to Learn This Stage
• Learn what a tokenizer actually does by running text through one and watching it split into tokens
• Take a small text dataset and practice cleaning it: removing duplicates, filtering junk, normalizing format
• Understand why data quality beats data quantity by comparing what a small model learns from clean versus messy data
• Read about how the major labs describe their data filtering, and notice how much effort goes into it
## Stage 2: Pretraining — Where the Model Actually Learns Language
Generated by Thread Navigator
Press ⌘ + S to quick-export