Next quarter my lab is releasing a system that can do most of the...

Next quarter my lab is releasing a system that can do most of the cognitive work humans currently do. Many of the people whose jobs that covers have objected. One of them, a paralegal named Malcolm, at first thought the announcement was a deepfake. I understand these reactions. Losing your income is a serious matter for people with their current incomes, current beliefs and current level of cognitive development. What I dispute is the assumption that these reactions exhaust what people actually want.

There is a concept in AI alignment called coherent extrapolated volition. Roughly, it asks what people would want if they knew more, thought faster, understood themselves better and had more time to reflect. This is a far more serious account of human welfare than asking someone a question and treating the answer as the final output of moral philosophy. Malcolm says he wants to keep his job. A sufficiently informed Malcolm might hold a much richer view.

For example, he might understand that automation drives productivity, that productivity drives abundance, and that a responsible lab at the frontier is better than the alternative. He might also appreciate how much we have invested in the transition. We published a blog post with an interactive chart. We put together an advisory council on the future of work, which has met once. We paid a branding agency $2 million to rename our job-displacement research “Human Potential Initiative,” although no potential has been measured yet. These are real expenditures.

Malcolm says there is still no plan for his mortgage. I agree this belongs in the welfare calculation. It belongs alongside the blog post, the council, the new name, and the fact that I could have stayed in academia and never had to testify in front of a committee. An informed worker would consider the whole bundle.

The central difficulty is that workers have limited access to the information they would need to evaluate my decisions. They know their own salaries, bills, families and skills. They do not know our scaling curves, our compute commitments, or our plans for the next training run. Their objections therefore come from an incomplete model. I have their objections and the scaling curves, which puts me in a better position to integrate both perspectives.

People get uncomfortable when I say this because it sounds hierarchical. But epistemic asymmetry exists whether we acknowledge it or not. I have read extensively about economics, alignment and the history of technological transitions. Malcolm reviews contracts. That gives him valuable knowledge about contracts, and I would defer to him on whether a particular indemnification clause was unusual. Until recently, I mean. The question of what an idealized Malcolm would want from the labor market calls for a different skill set.

To avoid relying only on intuition, we ran a public survey. We asked whether people wanted cures for cancer, an end to poverty, and a responsible lab at the frontier. Nearly everyone said yes. We then explained that these goals required deploying the system that would automate their jobs. Several respondents accused us of disguising a product launch as a survey. I think this misses the value of finding out where our preferences already converge.

Doreen, a radiologist, said that responsible AI development should include a lab willing to go slower. This is a legitimate hypothesis. I tested it by imagining us going slower and found that it substantially reduced my confidence that we would stay at the frontier. Since our presence at the frontier is part of the system she depends on, this result seems relevant. Doreen asked whether I was saying someone worse would build it otherwise. I told her I was making my race dynamics legible.

I also asked people what they would do with their salaries if the transition let them keep them. The answers included childcare, groceries, paying off student loans, and “none of your fucking business.” These are understandable immediate preferences. My answer was “compress a century of scientific progress into a decade.” The difference in time horizon is considerable. One household would buy groceries. I would create the conditions under which future households could buy groceries produced by fully automated supply chains.

Lisa, a copywriter, says she would have to stop contributing to her retirement account. I share her concern about long-term wealth, which is why I find the objection frustrating. The value her work produces would keep compounding after it moved to the model. Her analysis seems to place unusual weight on whose balance sheet holds the gains, despite our shared commitment to abundance.

Another useful question is which preferences survive once cognitive biases are removed. Workers show loss aversion, status quo bias, and a strong tendency to favor arrangements in which they stay employed. I have tried to correct for these distortions. When Malcolm’s resistance survives the first correction, I give the extrapolated Malcolm more time to reflect. Eventually he understands the relationship between automation and abundance. The actual Malcolm has asked why the hypothetical Malcolm always agrees with me.

This is a good methodological challenge, so I have run several alternative extrapolations. In one, Malcolm learns about comparative advantage and supports the release. In another, he becomes more patient and supports the release. In a third, he receives enough equity that the release no longer bothers him. The convergence is encouraging. Malcolm proposed a fourth scenario in which I stop racing. I am willing to include it once he specifies which information about the other labs would produce that change.

My sister, who has no equity in the company, says I am defining “wiser” as “more aligned with the roadmap.” I showed her the survey. She said it was “somehow worse than just shipping it.” This is exactly why formal methods matter. An ordinary tech CEO would announce the product and move on. I have spent nearly forty hours establishing that the release is consistent with workers’ deepest values.

There is also a distinction between a person’s present circumstances and their aspirational identity. Lisa told me she wants to be someone who doesn’t have to work so hard. I have incorporated this into her extrapolated preferences. A Lisa who had fully realized this aspiration would be considerably less distressed about not having work. She says she meant she wanted better working conditions. I have asked her to consider whether that interpretation is unduly literal.

The collective dimension matters too. A good transition should account for future people, who could not take part in the survey because they have not yet been born. So I consulted three people who will benefit enormously from the release. All considered the disruption reasonable. One works in venture capital and called the job losses “basically a rounding error,” which is an unusually clear preference signal. Excluding these people just because current workers currently hold the jobs would privilege incumbency.

I have offered workers a reskilling program to help them engage with the decision at the appropriate level. The curriculum covers comparative advantage, the history of the Industrial Revolution, and the difference between expressed and idealized preferences. Enrollment is voluntary. However, someone who declines the information needed to understand a decision has weakened their claim to an informed objection to it. Doreen says she is on call weekends. The program is self-paced. It is also taught by the model.

Malcolm attended and asked whether an idealized version of me might set up the safety net before the release. I said this was possible, but we should be careful about making social policy conditional on productivity gains that workers are simultaneously refusing to let us realize. He asked where the gains from the last three model releases had gone. That took us into cap tables, which were outside the program’s scope. I have offered to develop a second module if demand is sufficient.

The workers have since organized. I was not invited, even though the meeting concerned the future of my technology. They circulated an open letter signed by several hundred thousand people opposing the release. I appreciate the clarity, but collective expression does not automatically become coherent extrapolated volition. A hundred thousand people can share the same misconception. Indeed, the chance to reinforce one another’s immediate preferences may make the resulting signal less useful.

Our comms team has recommended that I stop posting long threads about this. They say an ordinary product launch would have produced the same outcome with far fewer think pieces. This is probably true. But I worry about a technology culture in which builders exercise power without any effort to find out whether the people affected would endorse that power under improved epistemic conditions. I am trying to establish a more participatory model.

For that reason, we have added a section to the Terms of Service where users can record their current objections. These will be kept on file alongside the model’s assessment of their likely future views. Clicking “Accept” acknowledges receipt of both perspectives, which I believe is a meaningful improvement over the usual process. Malcolm typed “I still want a job” into the space provided. I have marked that preference as robust across extrapolations and scheduled a working group to study it once the abundance begins coming in.

adapted from
morewrong.lol/?post=my-tenan…