Breaking production

The following is a conversation with Michael Paulson, better known online as ThePrimeagen. He is a programmer who has entertained and inspired millions of people to have fun building stuff with software, whether you're a newbie or a seasoned developer who has been battling it out in the software engineering trenches for decades. In short, ThePrimeagen is a legendary programmer and a great human being with an inspiring roller coaster of a life story. This is the Lex Fridman Podcast. To support it, please check out our sponsors in the description. And now, dear friends, here's ThePrimeagen.

What do you love most about programming? What brings you joy when you program? I can tell you the first time that I ever felt love in programming, or felt that joy or that excitement-

I would say the biggest one that I can really hold on to had to be one of two experiences. The first experience was when I was at a place called Schedulicity, and... Am I not allowed to say the place [inaudible 00:10:43] You're allowed.

Okay. You stream about all kinds of programming, but what kind of programmer are you? Are you full-stack developer, web programming? And maybe can you lay out all the different kinds of programming and then place yourself in that, in terms of your identity. Sexual identity as well. Yeah, we can put it all in there.

All right. Baby Prime. So you mentioned Netflix.

Let's try to reverse engineer some of the pain and some of the triumph. You made it sound easy at times. Let's try to understand it better, maybe when you were seven years old. What do you think about the pain you've experienced there, losing your dad? What do you think? What kind of impact did it have on you? What kind of memories do you have at that time? The best way I can put it is that I just never knew what a dad was. I was young enough that I could maybe repress or just even have the capability of remembering things long-term. Because I know most people don't remember a lot from when they're young, and so I'm not exactly sure. I probably was at one of the best possible ages, if I'm going to lose a dad, to lose a dad. You know? If you're going to lose one, if you're 11 or 12, it's a terrible age. That's what my brother was, and he fell into drug addiction and never got back out. So I just have more of a fuzziness and just kind of a longing. I just wish I had a dad.

I think a lot of people listening to this will resonate with your experience in high school. Being the outsider, being picked on, struggling through a lot of different complexities at home. What advice would you give to them? The worst part about high school is that you're surrounded by a bunch of people your age and it feels eternal.

Let me say it this way, some addictions have a much bigger societal look and porn is just not one of them, which makes it super hard. None of your friends are going to cheer you on. If you go on Twitter and say, "I quit porn," they're going to be like, "Well, that's good for you but not everybody..." No one makes that argument with meth, right? No one's going to be like, "Well, not everyone has to quit meth, okay. It's actually a fine industry and people who are the ones producing it, they're good also, right?" No one's going to make that kind of argument.

Can we try to reverse engineer that moment of you finding God. What is it at 19? Because it feels like that was a big leap for you to escape the pain, to escape the addiction or the beginning of that journey. What do you think happened there? I think it just felt like I just... There was no line that I wasn't willing to cross. Everything was fine and just like... It just all of a sudden, just in that moment, it's just like I had I guess some sort of deep fear and understanding I am going down a path. Is this really the path you want to go down?

The other moment you mentioned that I think is really inspiring is that you failed pre-calculus. You really struggled in school. You realize that school is really hard and then eventually you're able to sort of persevere and, I don't know, break through that wall of struggle. Can you, by way of advice, figure out what happened and what kind of advice you can give to people who are struggling? Yeah. I'll paint it in a more clear picture, or a very fast speed run of it is that I took pre-calculus, failed. I took pre-calculus again, failed, took pre-calculus again and got a C. So I took it three times. Then I took Calc over the summer, so Calc 1. In that one at the end, the final... The final was a two-hour final. I finished it in 30 minutes and that was the highest score in all of the school. And I proceeded to be the highest score in all calculus and Diffy Q.

For people who don't know, you worked in Netflix. By the way.

What was the technical learning curve for you? You showed up in your plaid pants, dressed up? Yeah.

Yeah. Can you actually speak to the print off debugging. You walk into a system and there's a lot of systems in the world like this. Twitter was like this, when Elon acquired Twitter and the rolls in and there's this old junky code base that's just like a giant mess, and you have to basically do print off debugging. What's the process of going into a code base and figuring out what the fuck? Well, how does this work? What are the flaws? What are the assumptions? You have to reverse engineer what all these other engineers did in the past and the mess across the space of months and years, and you have to figure out how all that works in order to make improvements. I've always just been good at print off debugging because one of my first kind of side quest jobs that I got was writing robots for the government when I was still at school. And so I'd kind of do this contractually for so many hours a week. And my boss, Hunter Lloyd, great professor by the way, he just said, "Hey, here's your computer, here's the robot, here's how you plug it in. Here's how you run the code. Can you write the flash driver, the ethernet driver. Can you write the planetary pancake motor? Here's some manuals, I'm missing some. Just figure it out, I'll be back." So that was government work for me. So I was like, okay, I'll figure all these things out. And I figured them all out and the only way to really get anything out of the machine was to print. And so it's like I had to become really good at printing my way through problems. And so that kind of became this skill I guess I adopted is that I can just kind of print off debug my way through a lot of these problems.

All right. Back to the wisdom of Reddit. Apparently there are memes and legends about your programming arc in Netflix. This Falcor system you mentioned, somebody, I think it was Teej, how do you pronounce his name by the way? Teej.

So in general, pushing to production, I talked to Peter levels about this. He, obvious he's operating as mostly a solo developer, but he often on the website said thousands, not hundreds of thousands of people use. He often ships to production, pushes to production, meaning just no testing, just like push to fix. What are the pros and cons of that approach in general to you? What do you think? It's obviously much easier the smaller your organization is. I think no one would argue that sentiment. If it's just you working on a singular project, it is obviously much easier for you to push directly to production because you are the only one working. You know all the ins and outs and if something were to break, you would discover it. So to me that makes sense. I think the way he operates is perfect for what he does. You couldn't take what he does and move it to say Microsoft or Netflix or Google because that would obviously... It would just be a disaster, just due to the amount of people all pushing to production. And so I personally love that. I think that you have to gauge both the application you're building and its complexity and what you're pushing, and how many people are working on it.

Let's just return to the infrastructure of the platform of Netflix and, speak more generally, Netflix, Twitch, YouTube. Anytime I use any of these services, I'm just blown away by the infrastructure it takes to deliver this service. YouTube and Twitch are unique, versus Netflix where the creators can roll in themselves and upload stuff. So on the consumption side, YouTube has over 100 billion views a day, over one billion hours watch time. But on the creator side, one million hours of videos are uploaded every day. One million hours. It's like you have to service both and you have to deliver everything... It's just incredible to me. Can you maybe speak to your own intuition, just zooming out on it, what it takes to deliver that kind of infrastructure? For me, the thing that I find vastly complicated and I can't imagine the engineering hours, is how do you even create an edge in that situation? And what I mean by an edge, when people say this phrase, if you're unexperienced, an edge is where you deliver data. You want that edge to be as close to the customer as possible because that's where the data lives. And then the communication between the customer and what you're doing is really, really small. Obviously the speed of light adds up, the amount of hops adds up, the amount of services that you have to remotely call adds up.

Okay. So yes, YouTube and Twitch and Netflix are doing an incredible job. You eventually chose, the madman you are, to leave Netflix and to start on a new journey of being a wolf pack of one, start streaming. What was that? What was the story of that? So I was streaming for almost seven years now. It started actually at Netflix. We did a charity, Extra Life, shout out to Extra Life for starting my streaming career, effectively is just you stream and whatever money you raise, it goes to Kids with Cancer research. They are a great charity in the sense that they take no overhead and they raise their own donations for their website and everything. And so it's a very great, straightforward charity. Really love what they've done. It was super cool because I live in South Dakota now, but I actually could choose a hospital directly where the money goes to. So there's a direct impact from A to B. So it's a pretty cool organization. And so my friend, Guy Cirino, Nice Try Guy is what I like to call him, he was probably the single greatest engineer I've ever met in my lifetime.

You do this incredible thing where you tried a bunch of different programming languages when you stream. You go all out on certain programming languages like Rust and then go and then try to pick a new one, but also are experimenting constantly. So maybe one question I could ask is about learning. What's your approach to learning a new programming language, and maybe what's your advice on learning a new programming language when you begin that journey? So I've kind of done a bunch of different ways to go through this learning process, and I've tried a lot of different ones. Something that is obviously successful is just start building something. Just put your hands on the keyboard, especially if you already know how to program. You're like, okay, I'm now using Zig. How do I do a main function so I can just run the program? Okay, now I know how to build. Okay, how do I do an if statement? What does it look like? Okay, how do I declare my own functions? How do I do modules, right? You just kind of Google your way through it, if you will, to get to the end product and build something. It's a great way to do things because I find that repetition, rote learning is obviously the best way to do this. You have to kind of go over it a bunch and you can definitely get out and build a lot of stuff with that. I like that initial kind of get used to things.

All right. Well, let me talk about the various languages. The cheesy, ridiculous question of what's the best programming language? Let's say, what's the best programming language that everybody should learn? Maybe let's go with the top five. I'm going to pull up the Stack Overflow developer survey, because I think we have... You don't like them? You got to remember, because I mean, you're a data guy. You know about biases and data. What does Stack Overflow naturally bias towards?

Also, we mentioned sort of in terms of applications, anything that relates to data or machine learning, Python is the sort of the leader there, so that's a great one. It seems like Python, CUDA stuff and C++ would be a dynamite in that, because a lot of these Python libraries are assumed you're just smuggling in C++ underneath the hood or C.

Do we count HTML and CSS as programming languages? I think there's some technical definition that it is. If you use this certain amalgamation of CSS plus HTML, it actually has, it can be a Turing complete language. But I mean for practical purposes, no. HTML is not a language. For me, yes, the Turing test is a good one, but for those that are just not wanting to be as academic, if I can't write a function and an if statement, I don't feel like that's... If I can't loop, if, and function, I don't feel like that's a good, that's a programming language.

I don't know. I kind of like these popular programming languages in this list. I like JavaScript. You like Bash?

Great account. There's certain Twitter accounts that are just legit. And I think FFMPEG, they have all these sort of parameters that you can add on the command line, that it's like one of those cryptic languages that only very few wizards understand. But once you begin to slowly understand, and I'm only at the very sort of beginning stage of that journey to mastery, the powers you gain at every step, it grows exponentially, it feels like. I mean, FFMPEG is just this incredible, what would you call a library system? There's just the people behind them must be just brilliant masterminds because they have to work with all these codecs, with all these containers, with all the mysteries of the media codec universe they're masters of. And they understand compression, which is another super fascinating technical set of problems that, I don't know, just FFMPEG just fills me with joy that it exists. But you need kind of Bash type comfort, command line comfort, to work with it to really unlock its power. Yeah.

Yeah. Actually, I saw Dave Plummer's testing of close to 100 languages for speed, and Zig came out on top. That was a mistake. I mean, when I say mistake, nothing against Dave Plummer. He's an extremely talented engineer. It's just that Zig, C, C++, all those languages that were being tested, they're all LLVM backends, right? That's the one that actually turns the thing into the executable part. And if there's a variation in speed, it just means in one language you didn't quite express what you are supposed to correctly. There's the language ball test that's been bouncing around on Twitter. Zig was like sixth or seventh below I forget what language it is. I played around with the example, added the word "no alias" to the argument, which means that the piece of memory that's coming into this function, there's no global pointers, there's nothing to it, and so the compiler can make these really cool optimizations. And I made it faster than the C version. So it just means that it's just not correctly specified is all that means.

You've programmed in Rust a lot. What do you love about Rust? What are the strengths? What are the weaknesses? Maybe you can speak about memory management that you already mentioned, the challenge of memory management that several of these languages address, but yeah, what do you love about Rust? What I love about Rust, I love the ability to free the memory that you're using is directly tied to the stack. So whenever you create something, there's a stack variable or there's some amount of stack memory, whether it's a pointer off to the heap, a pointer and a length. So some amount of memory on the stack and then some memory on the heap because a string is not all on the stack, it's some on the heap, some on the stack. And when that stack variable goes out of scope and gets cleaned up, it also cleans up what's on the heap. So it kind of simplifies this whole idea of, whoops, I forgot to free my memory. It just does it for you.

What are some epic projects you've built on stream that jump to memory? My most favorite, sorry for interrupting you. So I'm really jazzed right now.

So first of all, just enjoying the thing you create part, yeah. About there you can sit back and see all the parts dancing together. For me, also debugging, you get to see the choices you make materialize as how easy it is to debug. I'm a big proponent, I think you've mentioned this in the past, I put asserts everywhere. No, you are the reason why I do that.

I think this is more applicable for single systems like robots and so on. You have struggled with ADHD. I think a lot of people are really inspired by the fact that you're able to be productive and flourish while having ADHD. How'd you overcome it? Well, there's a lot of things that ADHD affects and so I'll start with some of the easiest things, because there's directly applicable, then these kind of collateral damage applicable things that happen. So one thing that has really helped me with ADHD is maturity. I think that's just a thing that everyone needs more of. Meaning that I found myself getting so wiggly and so out of control when I would try to sit down and read, and I just couldn't handle it. I just felt like I'd read a page and didn't read anything. The part of me that just went, "Oh man, gosh, I just can't even do this." I had to just simply quit listening to it and said, "Nope, I'm rereading this page." I remember reading some pages in college like 18 times in a row, just like I'm going to force myself to just do this the correct way.

Yeah, and that's the skill. What can you say to the skill of achieving focus in programming? Do you have a process of how you sit down and try to sort of approach a problem? So, all the different, not just distractions but the challenges of starting a project, of thinking through the design, how to maintain real focus, because it's really difficult intellectual endeavor. At this point I'm lucky, but when I first started I can remember that every last part of programming. I had to go look up, I had to go read, I had side quests at all time. Every step was a side quest. Why is my screen blinking when I'm trying to render this thing out? Oh, I didn't know about double buffering. Why is this happening? How do I even write to the screen? How do I... Everything was a question. I had more questions than answers. And so I constantly had the problem of side quests, and I find that to be a very exhausting thing. But as I learned my instrument very, very well, I don't have as many side quests. I become more and more able to just focus on the thing I want to do. And I find that to be something that is just super, super useful. So, when I say I'm kind of lucky, meaning that I've spent so much of my life preparing for this moment that now when I have the opportunity to do something, I can just do that thing and I don't...

All right, zero to 60, let's talk about the perfect setup. What's your perfect programming setup, keyboard operating system, how many screens, chair? All right, I like all these.

Yeah. Yep. I should mention that one of the things when I first became a fan of yours, I heard you talk about coffee and term... I still don't, by the way, understand what you're even talking about. I need to actually use it. But you run, amongst many things, a coffee company. Man, this smells so good. This one is dark mode, dark roast, whole coffee beans. There is seg origin, dash, dash location. Brazil. Yeah, there's a bunch of stuff on there.

All right. You've been both positive and negative on the role of AI in the whole programming software engineering experience. As it stands today, what do you think? What's your general view about AI? What is it effective at? What is it not so good at?

So given all that, can you give advice to young programmers? This is another question from Reddit, the infinite wisdom of Reddit. "What should a person in their early 20s do to move forward in the tech industry?" And this is an interesting addition to the question, "And by doing it, will this be walking on someone else's path?" I am going to try to answer that question, I guess the best I can, which I think that if you're entering into the tech world, one of the hardest pieces of advice that I took a long time to learn was I became enamored and addicted. Obviously, we talked about that I program for way too many hours, forgetting to spend the time I needed with my wife, with my friends, all that stuff, like totally wrapping myself up into one activity. I think though it made me who I am, it was probably an unhealthy activity and probably not a wise activity. And so the best advice I can give is that you got to develop the love, the skill, the desire for it. Whether that's just only using AI agents, programming yourself, using Zig or programming JavaScript, whatever that flavor is that's going to get you coming back every single day, getting the reps in the gym, if you will for programming.

Let me ask some random questions from Reddit. On a scale of one to 10, how much do you hate every product Microsoft has ever created and why is it a 10? I think we covered that. We haven't technically covered it.

Caulk, just to be clear. And try to avoid eye contact the whole time. You said that God was a big part, was a big part of your life. Can you speak to that a little bit more? Who is God and what effect, what role did he play in your life? So I did talk about that one important evening where I, for whatever reason, gained my conscious that moment. So obviously for me that I grew up with a life where I would probably argue myself as a functional atheist. I went to church a handful of times. I can't quite really remember actually going to church as a family in any sort of sense. So there wasn't some super strong tie or anything like that to it. Pretty much anyone else growing up in America in the '90s, you had some sort of impact or intersection with church at some point in your life, that was just a very normal thing I would probably say. And so when that happened, it was a fairly big surprise for me. I wasn't necessarily going that direction or deciding to do any of those things.