Contents

• Prizes for Posts
• The Review Vote Prizes
• How were prizes determined?
• The Unit of Caring prize pool
• Prizes for Reviewers
• One More Thing... The third annual review is complete. We spent two months nominating, reviewing, and voting on the best posts of 2020. After lots of thoughtful evaluation, it’s time to award some prizes! As described in the voting results post, there are two prize pools for posts. Lightcone Infrastructure contributed $10k to the Review Vote prize pool, and LessWrong users donated $1770 to the Unit of Caring prize pool (which Lightcone matched 1:1, bringing it up to $3540). Lightcone Infrastructure is also awarding $3600 in prizes for people who wrote reviews[1] for posts.

Prizes for Posts

The Review Vote Prizes

We’ve awarded a $1000 Top Prize to each of the following authors:

• catherio, for microCOVID.org, a tool to estimate covid risk

• Ajeya Cotra, for Draft report on AI Timelines

• evhub, for An Overview of 11 Proposals for Building Safe Advanced AI

• johnswentworth for When Money is Abundant, Knowledge is the Real Wealth, Alignment by Default, The Pointers Problem, and Coordination as Scarce Resource We’ve awarded a $500 Honorable Mention to each of:

• Alkjash, for Pain is not the unit of effort

• Mark Xu, for The Solomonoff Prior is Malign

• Jacob Falkovich, for Seeing the Smoke

• Alex Flint, for The Ground of Optimization

• Zvi, for Simulacra Levels and their Interactions

• Richard Ngo, for AGI Safety from First Principles

• Paul Christiano, for Inaccessible Information

• Steven Byrnes for My Computational Framework for the Brain

• abramdemski for Most Prisoner’s Dilemmas are Stag Hunts; Most Stag Hunts are Schelling Problems, and An Orthodox Case Against Utility Functions

• Daniel Kokotajlo, for Cortés, Pizarro, and Afonso as Precedents for Takeover

• Scott Garrabrant, for Introduction to Cartesian Frames

• Rafael Harth, for Inner Alignment, Explain Like I’m 12 Edition

How were prizes determined?

The LessWrong moderation team looked over the results of the vote, sliced four different ways:

• Total votes from all users;

• Total votes from 1000+ karma users;

• Total votes from Alignment Forum users;

• Weighted votes from *all users, but with 1000+ karma users weighted 3x. * That gave us some sense of how different users related to posts, and what they got out of them. In all cases, approximately the same posts showed up at the top of the rankings, albeit in somewhat different orders. I’m excited to reward all of them prizes for contributing to important intellectual progress. Two things stuck out between the various rankings:

• In the Alignment Forum vote, Draft Report on AI Timelines won by a landslide.

• In the "All Users" vote, the MicroCOVID.org announcement won by a landslide. Thoughts on the microCOVID.org Announcement Several people asked, upon seeing microCOVID.org in the review: "Is this what the Review is meant to reward? It sure was useful, but was it ‘intellectual progress’? Was it ‘timeless’? What would it mean to put it in a book?" We haven’t yet made our final call on whether/​how to do books this year, but if I were to print microCOVID, I’d specifically be interested in printing the whitepaper. Unlike the announcement (which would be quite weird to read in a book), the whitepaper is clearly a piece of enduring intellectual labor. I actually took some pretty timeless, valuable lessons from microCOVID – the schema that you can break "risk" into units that linearly add up was helpful for my overall thinking, and I expect to be relevant in other domains. And the conceptual breakdown of splitting up "person risk" and "activity risk" was useful. I think microCOVID was a great instance of applied rationality that built off a lot of concepts in our ecosystem, and translated them into a concrete product. Thoughts on Draft Report on AI Timelines Ajeya’s Draft Report on AI Timelines inspired a ton of discussion: Eliezer wrote a response post, and Holden wrote a counter-response, and there was further discussion on each post. I plan to include both follow-up posts as contextual reviews in the Best of 2020 sequences and/​or books. Daniel Kokotajlo wrote this review articulating why the post seemed important to him (edited slightly for brevity):

Whenever people ask me for my views on timelines, I go through the following mini-flowchart:

1. Have you read Ajeya’s report? – If yes, launch into a conversation about the distribution over 2020′s training compute and explain why I think the distribution should be substantially to the left, why I worry it might shift leftward faster than she projects, and why I think we should use it to forecast AI-PONR instead of TAI. – If no, launch into a conversation about Ajeya’s framework and why it’s the best and why all discussion of AI timelines should begin there. So, why do I think it’s the best? In a nutshell: Ajeya’s framework is to AI forecasting what actual climate models are to climate change forecasting (by contrast with lower-tier methods such as "Just look at the time series of temperature over time /​ AI performance over time and extrapolate" and "Make a list of factors that might push the temperature up or down in the future /​ make AI progress harder or easier," and of course the classic "poll a bunch of people with vaguely related credentials.") Ajeya’s model makes only a few very plausible assumptions. This is underappreciated, I think. People will say e.g. "I think data is the bottleneck, not compute." But Ajeya’s model doesn’t assume otherwise! If you think data is the bottleneck, then the model is more difficult for you to use and will give more boring outputs, but you can still use it. I think a lot of people are making a mistake when they treat Ajeya’s framework as just another model to foxily aggregate over. "When I think through Ajeya’s model, I get X timelines, but then when I extrapolate out GWP trends I get Y timelines, so I’m going to go with (X+Y)/​2." I think instead everyone’s timelines should be derived from variations on Ajeya’s model, with extensions to account for things deemed important (like data collection progress) and tweaks upwards or downwards to account for the rest of the stuff not modeled.

The Unit of Caring prize pool

Twelve users donated to the Unit of Caring prize pool. $1375 was donated to thank specific authors, and $395 was donated to the "general moderator discretion" fund. Lightcone Infrastructure matched all the donations, and the LessWrong moderation team reviewed and allocated them as follows:

• $700 to catherio for microCOVID.org, a Tool for Estimating Covid Risk

• $500 to Zvi for The Road to Mazedom, and Covid-19: My Current Model

• $420 to Johnswentworth for When Money is Abundant, Knowledge is the Real Wealth, and Alignment By Default

• $300 to Duncan Sabien for the CFAR Participant handbook

• $250 to Daniel Kokotajlo for Against GDP as Metric For Timelines

• $250 to abramdemski for Most Prisoner’s Dilemmas are Stag Hunts, Most Stag Hunts are Schelling Problems

• $250 to Jeffrey Ladish for Nuclear War Unlikely To Cause Human Extinction

• $140 to Mark Xu for The First Sample Gives the Most Information

• $140 to Richard Korzekwa for Why Indoor Lighting is Hard To Get Right and How To Fix It

• $100 to evhub for An Overview of 11 Proposals for Building Safe Advanced AGI

Prizes for Reviewers

The LessWrong Review isn’t just about posts. It’s also about the process of reflection and evaluation of those posts. We’ve been awarding prizes for reviewers who contributed significant commentary on posts. The totals came to:

• $600 to AllAmericanBreakfast

• $500 to johnswentworth

• $400 to Vanessa Kosoy

• $200 to Steven Byrnes

• $200 to Zvi

• $100 each to abramdemski, adamzerner, CharlieSteiner, Daniel Kokotajlo, Davidmanheim, Elizabeth, magfrump, MondSemmel, Neel Nanda, niplav, nostalgebraist, philh, Richard Korzekwa, Turntrout, Vika, Yoav Ravid, and Zack_M_Davis I separately hired Bucky to do some in-depth reviews. There were a few different ways I found reviews valuable. First, often they simply reminded me a good post existed, and gave some context for why it mattered. (johnswentworth’s comments on Subspace Optima were a short and sweet version of this. Daniel Kokotajlo’s comments on 11 Proposals for Safe Advanced AI were also a great example, which I expect fed into that post ranking highly in the overall vote) Second, there were many self-reviews by authors who had learned a lot. Of these, some of my favorites are:

• Steve Byrnes on Inner Alignment in Salt Starved Rats, where he exhaustively goes over some updates he made to his model.

• Niplav’s self-review of Range and Forecasting Accuracy (he expressed worry about a number of mistakes in the post, but I think the review is a great time to fix mistakes on otherwise good posts, and I was generally quite impressed with his thoroughness and thought the topic was quite important). Third, critical reviews.

• I thought TurnTrout’s comment on Nuclear War Is Unlike to Cause Human Extinction was succinct, and helped draw attention to a missing piece of the story. Landfish responded with some clarification.

• Bucky’s commentary on Kelly Bet On Everything helped iron out where the math exactly worked.

• AllAmericanBreakfast dug into the paper Scott cites in Studies on Slack, which taught me some new empirical things about the world as well as feeding into a comprehensive critique. Fourth, in-depth reviews that explored an idea and thought about its applications.

• Johnswentworth and Vanessa both had good reviews of The Solomonoff Prior is Malign, which is a concept I’ve found confusing to think about. My sense is that reviewing generally feels less glamorous than writing top level posts, but I think it’s quite important for grounding the overall process. To all our reviewers, thank you!

One More Thing...

That’s almost it for this year’s review. We have at least one additional bit of work, which is assembling the top posts into sequences. The winning posts will be integrated into our site library, promoted heavily for new users to read and reflect on. Congratulations to the winning authors – thank you for contributing to LessWrong’s longterm intellectual progress.

• ^During the Review Phase, people were encouraged to write comments reviewing each nominated post, reflecting on why it was valuable or how it could be improved.