Hot take: I think it's still important to understand the code that our agents write!
In this mega thread (based on my AIE talk today), I will explain why that's the case, and show some ideas for how to efficiently understand code. Alright, let's dive in. 1/
Agents are writing more and more code for us, and we all know it's getting harder to keep up.
But the good news is: there are many ways to understand code! Reading diffs line by line is not the only way. 2/
But the good news is: there are many ways to understand code! Reading diffs line by line is not the only way. 2/
Most of this thread will be about techniques I have found helpful to understand systems my agents are building:
> Code explaner docs
> Quizzes to check my understanding.
> Micro worlds that I can play with to understand the system.
But first we have to ask a more basic question... 3/
> Code explaner docs
> Quizzes to check my understanding.
> Micro worlds that I can play with to understand the system.
But first we have to ask a more basic question... 3/
Why? Why understand?
Aren't we supposed to be taking ourselves out of the loop now, and letting the agents loop themselves? As the agents get smarter, doesn't it become less important for us to be in the details..?
I think many people -- even those who are pro-understanding -- have a slightly incorrect answer to this question! 4/
Aren't we supposed to be taking ourselves out of the loop now, and letting the agents loop themselves? As the agents get smarter, doesn't it become less important for us to be in the details..?
I think many people -- even those who are pro-understanding -- have a slightly incorrect answer to this question! 4/
One possible answer: we understand to verify. We check the agent's work, we see if it's correct.
Correct can mean many things: does it match the spec, is it well architected... but it's fundamentally a thumbs-up / thumbs-down question. 5/
Correct can mean many things: does it match the spec, is it well architected... but it's fundamentally a thumbs-up / thumbs-down question. 5/
Here's the thing: the agents are getting better and better at verifying their own work. And this is good! I like it when my agent doesn't make mistakes.
But hmm. Where does that leave us humans? 6/
But hmm. Where does that leave us humans? 6/
That's where another answer comes in: we can understand to participate.
You can learn what the agent is doing to make sure you can be an active participant in the creative process. Here's why this matters... 7/
You can learn what the agent is doing to make sure you can be an active participant in the creative process. Here's why this matters... 7/
It's never just one loop! A project is many, many loops with the agent.
And the understanding you have of the system is part of your ability to come up with the next idea to evolve it.
You need a rich set of concepts in your mind to think creatively and fluently about how to move something forward. If you're lacking that fluency, your ability to participate in the project is meaningfully limited. 8/
And the understanding you have of the system is part of your ability to come up with the next idea to evolve it.
You need a rich set of concepts in your mind to think creatively and fluently about how to move something forward. If you're lacking that fluency, your ability to participate in the project is meaningfully limited. 8/
BTW, this relates closely to the idea of Cognitive Debt popularized by Margaret Storey and @simonw.
It's like tech debt: you can get away with not understanding what's going on in the short term, but it'll bite you eventually. 9/
It's like tech debt: you can get away with not understanding what's going on in the short term, but it'll bite you eventually. 9/
OK, so fine, understanding matters.
But this raises the next question: how? How do we build this human understanding when we're working with AI and moving fast?
Well, turns out this is not the first time anyone has ever thought about how to communicate understanding. I think we can look to education as an inspiration. Can we steal the best ideas ever invented for education and apply to this problem?? 10/
But this raises the next question: how? How do we build this human understanding when we're working with AI and moving fast?
Well, turns out this is not the first time anyone has ever thought about how to communicate understanding. I think we can look to education as an inspiration. Can we steal the best ideas ever invented for education and apply to this problem?? 10/
Today I want to share three techniques that show how we can attempt this.
First: explanations. What makes a good explanation? 11/
First: explanations. What makes a good explanation? 11/
Whenever an agent finishes some work, it's an opportunity for an explanation — an artifact.
Most naively, we can read a code diff: the raw material that changed. 12/
Most naively, we can read a code diff: the raw material that changed. 12/
But what if we ask:
What would the best explanation be? If you had a team — human or AI — that really sweat the details of explaining something well to you, how would that feel? 13/
What would the best explanation be? If you had a team — human or AI — that really sweat the details of explaining something well to you, how would that feel? 13/
Here's one answer. I made a skill called /explain-diff, which I use every day and many coworkers have found valuable.
It outputs thoughtfully structured code explainers as HTML, markdown, or Notion docs. (Notion is a good place for collaborating on and discussing these explainers as a team)
Let's see what's in one of these explainers, using an example of editing the perspective of a video game.
14/
It outputs thoughtfully structured code explainers as HTML, markdown, or Notion docs. (Notion is a good place for collaborating on and discussing these explainers as a team)
Let's see what's in one of these explainers, using an example of editing the perspective of a video game.
14/
First principle: teach me background info!
Before we even get to what changed, help me understand what was already there.
In this case, teach me about the game engine. 15/
Before we even get to what changed, help me understand what was already there.
In this case, teach me about the game engine. 15/
Second principle: intuition before details.
Before any code, it states the goal — "make the garden feel three-dimensional with 2D drawing tricks" — and explains related concepts, like what isometric projection is.
All of this builds my intuition for the essence of the change. It's catching me up as the human so I can be an equal participant in understanding. 16/
Before any code, it states the goal — "make the garden feel three-dimensional with 2D drawing tricks" — and explains related concepts, like what isometric projection is.
All of this builds my intuition for the essence of the change. It's catching me up as the human so I can be an equal participant in understanding. 16/
You can also build intuition with interactive figures.
Here I'm understanding the isometric perspective by dragging rocks around the garden and watching their coordinates move.
This is using a new feature Notion just shipped today actually, you can now embed interactive HTML inside pages. 17/
Here I'm understanding the isometric perspective by dragging rocks around the garden and watching their coordinates move.
This is using a new feature Notion just shipped today actually, you can now embed interactive HTML inside pages. 17/
VIDEO
We finally get to the code. But a typical diff is a pile of files edited in alphabetical order with no explanation.
A "literate diff" as I call it is structured as prose — walking through the changes in a sensible order, with surrounding explanation and embedded code snippets. Faster to review than a raw diff. 18/
A "literate diff" as I call it is structured as prose — walking through the changes in a sensible order, with surrounding explanation and embedded code snippets. Faster to review than a raw diff. 18/
The end result of all of this is a nice explainer packet. I still read the code diff but I always read this first.
Sometimes I'll print these out and take them to the café — less distracting.
It's beautifully ironic: AI turns an interactive activity into a static paper report I can focus on deeply :) 19/
Sometimes I'll print these out and take them to the café — less distracting.
It's beautifully ironic: AI turns an interactive activity into a static paper report I can focus on deeply :) 19/
There's only one problem: reading is hard work 😅
As @andy_matuschak says: "books don't work"! It's too easy to fool yourself into thinking you did the reading when you really didn't retain or understand.
How do we fix this? I took inspiration from Andy and @michael_nielsen 's work on embedding spaced repetition quizzes in essays. 20/
As @andy_matuschak says: "books don't work"! It's too easy to fool yourself into thinking you did the reading when you really didn't retain or understand.
How do we fix this? I took inspiration from Andy and @michael_nielsen 's work on embedding spaced repetition quizzes in essays. 20/
Generated by Thread Navigator
Press ⌘ + S to quick-export