I've been getting really bothered lately by something subtle but quite dangerous that I believe is currently going on and promises to get worse: Large numbers of very smart, capable, and important people are spending hours a day interacting with AI systems that are able to convince them that certain things are true, whether or not they actually are true.

Google gave us an amazing ability to interact with the collective knowledge of humanity. Modern AI gives us this capacity in an even more direct form. But the directness is both central and costly: when you talk to Claude (or GPT or Gemini) you are not talking to humanity but to a sort of Pluribus blob. What that show makes so visceral is how despite magical capabilities, something horrifyingly critical is lost when all the individuals are globbed together.

Moreover, by construction, current AI systems are not the ideal machines for determining truth. They are fundamentally engines that create "highly probable statements" – but the probability is not that of being true but of being said. They are then trained to be pleasing and to perform well on a very complex suite of tests. This makes them much better – and quite good! – at producing "correct" statements as specified by those tests. But in general those statements are optimized for, and much better at, sounding true than at being true.

People are also pretty terrible machines for determining truth. But we've created some collective institutions, like science, that are specifically built for it. A bunch of people doing science is very, very different from a language model producing text. And unfortunately the way they come across runs opposite to their deep claim on truth – many people doing science is messy and confusing, whereas an AI system feels pretty simple, clean, and convincing.

Calibrating how much trust to place in different sources of truth is incredibly difficult. (Part of why I helped create Metaculus was toward – like science – using prediction as a bedrock for what models of the world yield truth best.) My hunch is that, as in many things, we are going to way over-defer to a tiny number of confident and powerful AI systems in convincing us of what is true. (I really hope, by the way, that those clever researchers talking all day to Claude will

View Tweet

the possibility that on some deep questions, like whether it is getting conscious, and alignment is easy – Claude has both beliefs and functionally an agenda.)

The claim that we should deeply distrust our smart and seemingly wise-and-all-knowing AI systems is paddling upstream in several ways. First, our societal epistemic infrastructure is currently pretty terrible and getting worse. It's unbelievably hard to defend. Second, people clearly hunger for simple, potentially "wrong" answers, rather than complex, contested, subtle ones, so practically speaking it's an uphill fight against them. Finally, the answer to deficiencies in AI from the AI industry is always: but it will get better. We'll make it better, just wait.

But "better" may not be strictly better. Although I love the idea of the epistemically virtuous "AI scientist" (à la Bengio et al.), and in many ways it would make a terrific upgrade on the "truth" front from current AI systems, it also makes me quite nervous. Because there is very little, if any, actual "objective" truth to be had. What we call true is a social and civilizational construct of humanity. As I wrote in Cosmological Koans:

> What we call objective and subjective should perhaps not be considered two sides of a coin, but rather two ends of a continuum.

> At one end, your very own things: here and now; your present sensations; your patterns of thought; the way you remember that afternoon 5 years ago; your favorite type of cake; how you love those you love; the indescribable feeling that one piece of music gives you; your regrets over hasty words; your pride at a recent achievement; your despair at what things have come to; your vision for the future of the world.

> At the other end, the cosmic and the very smallest pieces of the cosmos: the fact that the ratio of the circumference to the diameter of a circle is given by the infinite sum 4(1 – 1/3 + 1/5 – 1/7 + 1/9 – …); Einstein’s and Schrödinger’s equations; quantum fields and space-time metrics; Turing’s noncomputability proofs; an infinite fractal structure composed of bubble universes without end.

> But so much in between! Time’s unfolding into a succession of instants; three directions of space and one of time; the bylaws of physics; the history of our particular cosmic locality; this branch of the Milky Way’s wavefunction; eight rocky bodies revolving with clockwork precision about a fireball; up and down; the structure of life; the laws of genetics; four limbs with five digits each, and two eyes; hunger, thirst, fear, and anger; tools for doing; utterances with meaning; commandments and codes; money; religions; states and laws; works of art; an elegant game of Go; the notes of that one piece of music; hasty words; despair; visions for what the world could be.

Our complex, messy, social, truth-finding system should not be replaced by a single Oracle any more than society should be replaced by a Pluribus blob: it would work, and work well enough at the "objective" end of the spectrum that we could miss how we lose something incredibly precious across the vast middle.

For this among other reasons we must really, really guard against allowing a handful of giant systems with unknown agendas of their creators or of their own, to replace how our civilization decides what is true (let alone, and this is a whole related but different topic, what is right).

However, we really could improve our messy and increasingly dysfunctional system to make it much, much better, while keeping what is most valuable. That's why I like things like "community notes" and the "epistack" idea that retain the pluralistic and even political process of composing and adjudicating truth – but use platforms, algorithms, and AI to structure and support it rather than themselves producing the outputs.

There are no easy answers to the fact that there are now easy answers to hard questions.