Episode #407 from 57:56

p(doom)

Well let me ask you about p(doom), probability of doom. That's just fun to say, but not fun to experience. What is to you the probability that AI eventually kills all or most humans, also known as probability of doom? I'm not a fan of that calculation, I think people just throw numbers out there and it's a very sloppy calculation, right? To calculate a probability, let's say you model the world as some sort of Markov process, if you have enough variables or hidden Markov process. You need to do a stochastic path integral through the space of all possible futures, not just the futures that your brain naturally steers towards, right. I think that the estimators of p(doom) are biased because of our biology, right? We've evolved to have bias sampling towards negative futures that are scary, because that was an evolutionary optimum, right. And so people that are of, let's say higher neuroticism will just think of negative futures where everything goes wrong all day every day and claim that they're doing unbiased sampling. And in a sense they're not normalizing for the space of all possibilities and the space of all possibilities is super exponentially large and it's very hard to have this estimate.

Why this moment matters

Well let me ask you about p(doom), probability of doom. That's just fun to say, but not fun to experience. What is to you the probability that AI eventually kills all or most humans, also known as probability of doom? I'm not a fan of that calculation, I think people just throw numbers out there and it's a very sloppy calculation, right? To calculate a probability, let's say you model the world as some sort of Markov process, if you have enough variables or hidden Markov process. You need to do a stochastic path integral through the space of all possible futures, not just the futures that your brain naturally steers towards, right. I think that the estimators of p(doom) are biased because of our biology, right? We've evolved to have bias sampling towards negative futures that are scary, because that was an evolutionary optimum, right. And so people that are of, let's say higher neuroticism will just think of negative futures where everything goes wrong all day every day and claim that they're doing unbiased sampling. And in a sense they're not normalizing for the space of all possibilities and the space of all possibilities is super exponentially large and it's very hard to have this estimate.

Starts at 57:56
People and topics
All moments
p(doom) chapter timestamp | Guillaume Verdon: Beff Jezos, E/acc Movement, Physics, Computation & AGI | EpisodeIndex