Reflections on Anthropic and EA

Personal reflections I've been sitting on lately.

These are personal reflections on feelings that I’ve been sitting with recently. I’m posting them, because the last time I felt this way I regretted not doing so. I don’t really know how calibrated they are, but I’ve been noticing them more and more.

Around 6 months before FTX collapsed, I wrote a draft EA Forum post called “Concerns about the Carrick Flynn campaign.” At the time, I was on the verge of leaving EA. During the frothiest FTX days it seemed like many folks were energized by the money and attention, but I felt kind of gross. The amount of money pouring into EA, and the resulting “lowering of the bar” that happened felt like a degradation of community norms that was too severe, almost not worth the benefits. Of course, some people voiced these concerns at the time, but they were drowned out by the excitement of the moment.

I never published the post, but I ended it by saying:

This feels incredibly poorly thought out, like it is one negative news article away from ruining EA’s reputation in the public’s eye, and that it also represents some degree of moral/value slippage that probably shouldn’t be tolerated by a community really focused on doing the most good. I admit that all of this has made me feel more disconnected from EA than I’ve ever felt before, despite also feeling very value-aligned with the people making these decisions.

I don’t really know what to do about all of this, but I guess I’m just scared?

I’ve noticed over the last two months that some of these same feelings are coming back for the first time since FTX. This time, I’m paying a lot more attention to them, because last time I think these feelings turned out to be very attuned to a risk the community was mostly ignoring.

For me, an essential feature of EA has always been its demandingness. I don’t mind that my commitment to utilitarianism pushes me to hold views that might be considered weird in mainstream circles. I don’t mind that EA can be all consuming. I don’t mind that EA can push me to make hard trade-offs for myself. We’re trying to make the best possible world. I would have been foolish to sign up while thinking it would be easy or pleasant. The EAs I like best are people who take the mandate of morality seriously. I disagree with many of these people’s views (I think we should pay more than the absolute minimum salary, I don’t think that our most fundamental goal should be to make the largest good future, etc.) but their beliefs are demanding, and I deeply respect people who follow that demandingness to the very end.

Before FTX, the EA I was in was more universally demanding. During the FTX era, it lost that for a while. Getting your project funded basically required that you had a pulse and sounded like an EA. Today, in response to a growing sense that major Anthropic money is coming, I am starting to have a sense of that same slippage happening again. As a result, when I reflect on how I respond to choices I’m given, I’m worried that much of what feels essential in me about being an EA is being corrupted again.

I hear daily about the coming Anthropic funds. Of course, Anthropic is very different from FTX. It is a legitimate company and has a reasonable chance at being the most important company in the world at a moment that may be one of the most important in history. Many of the people I’ve met who work there are thoughtful, generous, charitable people. I don’t think anyone is committing an enormous fraud, and I think the work they do is fundamentally much more important than FTX’s.

But at the same time, I’ve had a growing number of experiences that make me feel like I’m back in the FTX era. Immense, undemocratic political spending is ramping up (though spending by the enemies of AI safety is also growing) — I have no idea if this is good or bad, but it is, for my sense of morality, unsettling. I have been offered jobs that are billed as impactful and that pay vastly more money than I need while not providing any clear benefit for the world. I feel daily pressure to move to San Francisco, where I’m told that I can “increase [my] influence over the future by somewhere between 10x and 1000x” (an extraordinary claim without extraordinary evidence). It is hard to make the world better, and we should expect it is nearly impossible to do it effectively. It shouldn’t feel like an easy choice to try. To move to San Francisco, and take a cushy job where I am paid an enormous amount of money to write navel-gazing think pieces, and spend all day in a lovely office with the smartest people in the world while eating free lunches, feels like an easy choice.

Right now, I’m leaving my job to go start a new charity, working on a very narrow intervention. Despite every comparison in potential impact I and others make on paper suggesting this is where I will have the most impact, I instead get constant, subtle and unconscious signals from the community that I’m making a major mistake: I’m turning down economic security by turning down jobs in “EA” that pay significantly more; I’m not moving to the place “where I’ll have the most influence,” etc. But when I look at some of the other options I feel fortunate to have, and try to imagine taking them if they either didn’t bring me closer to power or paid only what I make now, they lose all appeal. That they appeal at all to me right now feels fundamentally like a corruption of my spirit. I want to be motivated by the impact of my actions, and I’m constantly receiving a subtle signal that I should put aside what is more obviously impactful, go make a lot of money, and get closer to a lot of power.

This is, of course, very tempting, and sounds very exciting. It would be a glorious coincidence if, of all possible things I could be doing, the thing that is most intellectually interesting and fun and increases my status the most also happens to be the most impactful thing I could do with my life. Similarly, why is it that taking a high-paying, high-power job where the promise of impact is a vague sense of “influence” at the cost of progress on a difficult, high-risk, and concrete intervention is so appealing?

Outside of EA, this pressure isn’t surprising — but that this pressure now feels like it is coming from many in EA, or affiliated with the communities I care about, is what confuses me. I think I used to find EA to be a bulwark that supported me through hard moral decisions — now, it feels more like a counterpressure to making the right choice. The most tempting opportunities to not do the most good I can are coming from the community itself.

In a sense, I feel a bit of my spirit being corrupted. I, and my community, are being given choices that I think will take moral courage to navigate, and I don’t know if we are always going to make the right choices. The community has money, and so it is growing. But I think a core lesson of FTX was that the growth that comes with being seen as “the people who will give money to everything that’s vaguely aligned” is growth that could be not worth having.

The most obvious reason that money and power would align with expected value is that, unfortunately, money does make it possible to do a lot of things in the world. I could be wrong to be worried. I generally do think this new, vast source of funding will probably be good, and I don’t have the same sense I had during the FTX era of there being genuinely massive corruption occurring. No one has tried to pay me $10k to fly to the Bahamas just to hang out (yet). Maybe the community becoming less pleasant for me is not really a sign that it’s becoming less rigorous, and just an indicator that I fit in less with the new trends. Many people in the Bay doing work with or against AI labs are doing incredibly difficult things that take a huge amount of moral courage — especially those directly challenging labs. But at the same time, I think there are also major costs to gaining power and influence — by and large, people seem to make worse decisions when they have them. I don’t think our community has figured out how to navigate these trade-offs as it steps closer to significant power.

As a result I worried I’m going to lose the EA I loved. When EA had no power and low impact on the world outside the people helped by bednets and the chickens living better lives, I had absolute faith in the community — the people I met were all my allies in this strange corner of ethics, who I trusted as strangers more than I trusted anyone else. Now, I regularly meet new people in the community who feel only like strangers, or worse, strangers who want to step on my shoulders to meet my richer and more powerful friends and connections. Now, many people I know are spending more time trying to influence Anthropic staff than figuring out how to do good and acting on it. Maybe it’ll pay off. I hope it does. But there is a part of me that is skeptical.

I think EA’s failure to grapple with the corrupting influence of power is among its greatest failures. As EA, or at least, a company that is strongly influenced by EA values and approaches, is once again at the top of the world, my faith that we’ll keep making the right choices is slipping away. As I said the last time I typed a post like this, I don’t really know what to do about all of this, but I guess I’m just scared?