Many attempts to do good fail, but the best are exceptional
- Page 6
-

Introduction to Effective Altruism
- Page 6
-

By remaining open to working on different causes, we’re able to change course to where we can make the biggest difference, without restricting ourselves too early.
- Page 8
-

The cause that you choose to work on is a big factor in how much good you can do. If you choose a cause where it’s not possible to help very many people, or where there just aren’t any good ways to solve the relevant problems, then you will significantly limit the amount of impact you can have.
- Page 8
-

useful.
- Page 8
-

US households spend around 2% of their budgets on personal insurance, on average. If we were to spend a comparable percentage of
- Page 9
-

Improving the long-term future
- Page 9
- h4,

Animal suffering
- Page 9
- h4,

Fighting extreme poverty
- Page 9
- h4,

Improvements to the scientific establishment, such as greater transparency and replication of results
- Page 10
-

Prevention of road traffic injuries
- Page 10
-

(highlight:: 1. Consider which problem you should focus on.
2. Consider the most effective way for you to address the problem)
- Page 10
-

US criminal justice reform
- Page 10
-

Other causes
- Page 10
-

Pledging to give
- Page 11
- h4,

Donating effectively
- Page 11
-

opportunity costs: buying one thing costs money that could be used to buy others
- Page 13
-

High overhead costs are only one possible failure mode for a charity. Consider again the Arctic explorer, trying to decide between a $200 parka and a $200 digital camera. Perhaps a parka only cost $100 to make and the manufacturer takes $100 profit, but the camera cost $200 to make and the manufacturer is selling it at cost. This speaks in favor of the moral qualities of the camera manufacturer, but given the choice the explorer should still buy the parka. The camera does something useless very efficiently, the parka does something vital inefficiently. A parka sold at cost would be best, but in its absence the explorer shouldn’t hesitate to choose the parka over the camera.
- Page 14
-

And likewise, there is only one best charity: the one that helps the most people the greatest amount per dollar. This is vague, and it is up to you to decide whether a charity that raises forty children’s marks by one letter grade for $100 helps people more or less than one that prevents one fatal case of tuberculosis per $100 or one that saves twenty acres of rainforest per $100. But you cannot abdicate the decision, or you risk ending up like the 11,000 people who accidentally decided that a pretty picture was worth more than a thousand people’s lives.
- Page 14
-

If a high-powered lawyer who makes $1,000 an hour chooses to take an hour off to help clean up litter on the beach, he’s wasted the opportunity to work overtime that day, make $1,000, donate to a charity that will hire a hundred poor people for $10/hour to clean up litter, and end up with a hundred times more litter removed
- Page 15
- impact,
- [note::EA in a nutshell - while valid, this kind of thinking can be incredibly toxic.]

The Roman historian Sallust said of Cato “He preferred to be good, rather than to seem so”. The lawyer who quits a high-powered law firm to work at a nonprofit organization certainly seems like a good person. But if we define “good” as helping people, then the lawyer who stays at his law firm but donates the profit to charity is taking Cato’s path of maximizing how much good he does, rather than how good he looks.
- Page 15
-
- [note::I love this phrasing - "Being good as opposed to seeming good"]

counterfactually
- Page 17
- belonging, community building, welcomeness,
- [note::Huh, I remember this word being very foreign to me when I first started becoming involved in EA. I should make a point to remember this feeling when talking to new people in the EA Philly community.]

When you see gold, take a moment to think about what you value. Many people won’t just value one particular thing. However, do think about what you care about and put that in place of the gold.
- Page 17
-
- [note::This seems like a useful exercise in terms of figuring out what top-level goals one should strive for.]

Elsewhere in this conference, there are going to be treasure maps and discussions of where the gold is. I’m not going to do that in this talk. I’m instead going to be focusing on the tools and techniques that we can use for locating gold, rather than trying to give my view of where it is directly.
- Page 18
-
- [note::I like this framing - EA is really all about finding out what and where the gold is ("gold" being what we altruistically value)]

So, on the left of the distribution that is not heavy-tailed, I can see that the gold is fairly evenly spread across a lot of different places. If we want to just get most of the gold, what is important is getting to as many different places as possible. Solar power is like this. Sure, some places get more sunlight than other places, but the amount of solar power you generate depends more on how many total solar panels you have, than on exactly where you place them. Over on the right, though, we have a distribution where you can see a lot of the area that spikes on the right-hand side. This just means that a lot of the gold, and if this is an actual stand-in for something that we value, a lot of what is valuable comes in this extreme of the distribution of things, which are just unusually good.
- Page 20
-

So just because somebody says, “Hey, this is gold,” doesn’t mean we should always take people’s word for it.
- Page 24
-

A nice property of literal gold is that when you dig it up, you’re pretty sure that you can recognise “yes I have gold”. We often have to deal with cases where we don’t have this. We don’t have the gold, so we have to carefully try to infer its existence, by using different tools. This fact is like the dark matter of value.
- Page 24
-
- [note::"Altruistic value is like dark matter - we have to infer it's existence by using different tools"]

You can still get gold, but it’s more work for each little bit, for each nugget that you’re getting out. This is the general phenomenon of ‘diminishing returns’ on work that you’re putting in. This concept comes up in a lot of different places, and so it is worth having an understanding of it.
- Page 25
- diminishing returns,

Running out of easy gold
- Page 25
-

That is great because we have taken those low hanging fruit. Or similarly, if in AI safety, writing the first book on superintelligence is a pretty big deal. Writing the 101st book on superintelligence is just not going to matter as much.
- Page 26
-

The first of these dimensions is scale. All else being equal, we would prefer to go somewhere where there is a lot of gold, rather than a little bit of gold.
- Page 26
-

Second: tractability. We’d like to go somewhere where we make more progress per unit work. Somewhere, where it’s nice and easy to dig the ground, rather than trying to get your gold out of a swamp.
- Page 27
-

And third is uncrowdedness. This has sometimes been called neglectedness. I think that term is a bit confusing. It’s a bit ambiguous because sometimes people use ‘neglectedness’ to mean that this is an area which we should allocate more resources to. What I mean here is that there aren’t many people looking at it. All else being equal, we would rather go to an area where people haven’t already picked up the nuggets of gold on the ground, than one where they have. And now the only gold remaining is quite hard to extract.
- Page 27
-
- [note::Nuanced yet important distinction between "neglectedness'" and "uncrowdedness]

The first term is measuring the amount of value you get for, say, solving an extra one percent of a solution.
- Page 28
-

It is an elasticity here, which is a technical term. It’s a pretty useful and general term (go look it up on Wikipedia, if you’re interested). Here, it is measuring, for a proportional increase in the amount of work that’s being done, what proportion of a solution that gives you.
- Page 28
-
- [note::#research-this]

The final term just cancels to one over the total amount of work being done. So that is very naturally a measure of uncrowdedness.
- Page 28
-

I’m not going to spend an hour helping a bee, even if nobody else is helping it and it would be pretty easy to help, because just the scale of it is pretty small. I don’t think we should work on perpetual motion machines, even though basically nobody is working on it and it would be really fantastic if we succeeded. Because it seems like it’s not tractable. And this [...] might give us a warning against actually working on climate change. Because at a global scale, that gets a lot of attention, as a problem.
- Page 29
- tractability, itn framework, importance, neglectedness,

Absolute and marginal priority
- Page 30
- 1action/research,

If we are having a conversation about what we as individuals or as small groups should do, I think it’s appropriate to use this notion of marginal priority, of how much do extra resources help. If we’re talking about what we collectively as a society or the world should do, I think it’s often correct to talk about this kind of notion of absolute priority and how much resources ought to be invested in it, total.
- Page 31
-
- [note::Important distinction]

And maybe we have some technologies which are destructive. So we can use dynamite and dynamite gets us loads of gold now, but it also blows up some gold, and now we never get that gold later. That could be pretty good if you are focusing just on trying to get gold in the short term. But it could be bad from this eventual gold perspective.
- Page 31
-

And so one of the major drivers of how much gold is eventually extracted is the order in which the technologies are developed, and the sequencing. If we discover the dynamite first, people are going to go and have fun with their dynamite and they’re going to destroy a lot of the gold. If we discover the drill first, then by the time dynamite comes along, people will go “Well, why would we use that? We have this fantastic drill.”
- Page 32
-

Comparative advantage
- Page 32
-
- [note::How can we, as a community, figure out how to effectively distribute our talents? Perhaps performing analytics on EA Hub cause area data?]

I have Harry, Hermione and Ron, and they have three tasks that they need to do to get some gold. They need to do some research, they need to mix some potions, and they need to do some wand work. Hermione is the best at everything, but she doesn’t have a time turner, so she can’t do everything. So we need to have some way of distributing the work. This is the idea of comparative advantage. Hermione has an absolute advantage on all of these tasks, but it would be a waste for her to go and work on the potions because Harry is not so bad at potions. And really, nobody else is at all good at doing the research in the library. So we should probably put her on this.
- Page 33
- coordination, comparative advantage,

This is a bit more speculative, but I think we can also apply this at the time level. We can ask ourselves, “What are we, today, the people of 2016, particularly well suited to do, versus people in the past and people in the future?” We can’t change what people in the past did. But we can make a comparison of what our comparative advantage is relative to people in the future. And if there is a challenge, if there were going to be some different possible challenges in the future that we need to meet, it makes sense that we should be working on the early ones. Because if challenges are coming in 2020, the people in 2025 just do not have a chance to work on that.
- Page 33
-

So, maybe it is the case that we can adapt one of these existing institutions for our purpose, which is trying to aggregate knowledge about what are the ways to go and do the most good. But maybe we want something a bit different, and maybe somebody in this room is going to do some work on coming up with valuable institutions for this. I actually think this is a really important problem. And it’s one that is going to become more important for us to deal with as a community, as the community grows.
- Page 35
-

Do you believe it because somebody else told you? Do you believe it because you have really thought this through carefully and worked it out for yourself? There is a blur between those. Often somebody tells you, and they give you some reasons. And you are thinking: “Oh, those reasons kind of check out,” but you haven’t deeply examined the argument yourself.
- Page 36
- deferring, favorite,
- [note::This absolutely applies to me. Though, it's hard to develop a well-informed opinion about something. That's why I'm so reluctant to do it.]

I’m shortening the chain of Chinese whispers, of passing this message along. And as things get passed along, it’s possible that mistakes enter or just something isn’t well grounded, and then it gets repeated. By going back and checking earlier sources in the chain, we can try to reduce that, and try to make ourselves more robustly confident in these statements.
- Page 37
-
- [note::"Shortening the chain of whispers" has risks associated with it (related to "value of information"?)]

Heavy-tailedness isn’t just a binary property; it’s a continuum
- Page 38
-
- [note::Not really sure what this means #research-this]

Heavy-tailed distributions
- Page 38
-

If people are pretty good at identifying the best opportunities, and they are uniformly seeking out and taking them, then the best things that are left might not be so much better
- Page 39
-

Ways to make money, maybe they actually start out distributed across a wide range. This is a log scale now, and it is meant to represent one of those heavy-tailed distributions, but then people who are losing money, say, “Well, this sucks,” and they stopped doing that thing. And they see other people who are doing activities which are making lots of money. And they think: “Yeah, I’m going to go do that.” And then you get more people going into that area, and then diminishing returns mean that you actually make less money than you used to, by doing stuff in that area.
- Page 39
-

Differential progress
- Page 42
-
- [note::#research-this]

Conclusion
- Page 44
-

crucial consideration
- Page 45
-
- [note::#research-this]

a crucial consideration is a consideration such that if it were taken into account it would overturn the conclusions we would otherwise reach about how we should direct our efforts, or an idea or argument that might possibly reveal the need not just for some minor course adjustment in our practical endeavors but a major change of direction or priority.
- Page 45
- crucial considerations,

deliberation ladder, which would be a sequence of crucial considerations, regarding the same high-level subgoal, where the considerations hold in opposing directions.
- Page 46
-

So a crucial consideration component will be an argument, idea or datum which, while not on its own amounting to a crucial consideration, seems to have a substantial probability of maybe being able to serve a central role within a crucial consideration.
- Page 46
-

technological completion conjecture — that in the fullness of time, unless civilization collapses, all possible general useful technologies will be developed —, these dangers will have to be confronted, and all our choice really concerns is the sequence in which we confront them.
- Page 48
-

Whereas with utilitarianism, the utilitarian preference extends far and wide beyond our familiar environment, including into the cosmic commons and billions of years into the future and super advanced civilizations: what they do matters from the utilitarian perspective, and matters a lot. Most of what the utilitarian preference cares about is stuff that we have no familiarity with.
- Page 49
-

if one tries to think about how to apply utilitarianism to a world that has a finite probability of being infinite, one will run into difficulties in terms of how to measure different infinite magnitudes and still seeing how we could possibly make any difference to it.
- Page 49
-

I think that these deliberation ladders are particularly likely to turn up when one is trying to be a thoroughgoing utilitarian and one really takes the big-picture question seriously.
- Page 49
-

The third possible reason here is that one might think that we are kind of close, not super close, but close to some pivot point in history. That means that we might have special opportunities to influence the long-term future now. And we’re still far enough away from this: it’s not obvious what we should do to have the maximally beneficial impact on the future. But still close enough that we can maybe begi
- Page 49
-

This could affect the personal domain as well. It’s just like with an ordinary person’s typical utility function: they probably don’t place a million times more value on living for a billion years than living for a hundred years, or a thousand times more value on raising a thousand children than on raising one child. So even though the future still exists, it just doesn’t weigh as heavily in a normal human utility function as it does for utilitarians.
- Page 50
-

50 P R I N C I P L E S to perceive some contours of the apparatus that will shape the future.
- Page 50
-

We do something analogous to that in other domains. Like a typical traditional public policy, social welfare economists might think that you need to maximize some social welfare function which might take a form like this.
- Page 51
-

If we plot capacity, which could be level of economic development and technological sophistication, stuff like that, on one axis and time on the other, my view is that the human condition is a kind of metastable region on this capability axis. You might fluctuate inside for a while, but the longer the time scale you’re considering, the greater the chance that you will exit that region in either the downwards direction and go extinct
- Page 51
-

Maxipok principle that Beckstead also mentioned: Maximize the probability of an OK outcome.
- Page 52
-

principle of differential technological development suggests that we should retard the development of dangerous and harmful technologies—once the raise existential risk, that is— and accelerate technologies that reduce existential risks.
- Page 54
-

The point here is that to have the best possible condition; we need super advanced technology: to be able to access the cosmic commons, to be able to cure all the diseases that plague us, etc. I think to have the best possible world, you’ll also need a huge amount of insight and wisdom, and a large amount of coordination so as to avoid using high technology to wage war against one another, and so forth.
- Page 54
-
- [note::This feels almost greedy (i.e. why can't we stay in our current situation?), but the reality is that we can't really hover - human progress is unlikely to stop because things like impending climate change and our current economic system will it to continue.]

However, we might feel it’s very difficult to get strong reasons for knowing whether more people in this room is better, or whether there is some inverse relationship. A good signpost would strike a reasonable compromise between being visible from afar and also being such that we can have strong reason to be sure of its sign.
- Page 55
-
- [note::I don't really understand the "being visible from afar" part]

Nietzschean Umwertung aller Werte — the revaluation of all values
- Page 57
-
- [note::Huh?]

To just pick an example: insects. If you are a classical utilitarian, this consideration arises within the more mundane—we’re setting aside the cosmological commons and just thinking about here on Earth. If insects are sentient then maybe the amount of sentience in insects is very large because there are very,
- Page 57
-

Take into account that expected value changes are probably smaller than they appear. If you are a utilitarian, let’s say you think of this new argument that has this radical implication for what you should be doing, the first instinct might be to radically change your expected utility of different practical policies in light of this new insight. But maybe when you reflect on the fact that there are new crucial considerations being discovered every once in awhile, maybe you should still change List of some areas with candidate remaining CCs or CCCs • Counterfactual trade • Simulation Stuff • Infinite paralysis • Pascalian muggings • Different kinds of aggregative ethics (total, average, negative) • Information hazards • Aliens • Baby universes • Other kinds of moral uncertainty • Other game theory stuff ` • Pessimistic metainduction; epistemic humility; anthropics • Insects, subroutine
- Page 58
-
- [note::Hmmm... this is good to keep in mind. "If a new insight has the tendency to radically change your perception in one direction, then it's not impossible that another new insight could radically change your perception in the opposite direction, and so your overall perception might not change as much as you think it will."]

very many of them. So that maybe the effect of our policies on insect well-being mind trump the effect of our policies on human well-being or animals in factories and stuff like that. I’m not saying it does, but it’s a question that is non-obvious and that could have a big impact.
- Page 58
-

59 P R I N C I P L E S your expected value, but not as much as it seems you should the first time.
- Page 59
-

Parliamentary Model
- Page 59
-

This is the idea that if you are unsure as to which moral theory is true, then you should assign probabilities to different moral theories and imagine that there were a parliament where each moral theory got to send delegates to that parliament in proportion to their probability.
- Page 59
-

The idea is that, other things equal, the more probability a moral theory has, the greater its say in determining your actions, but there might also be these trades between different moral theories
- Page 59
-

So if you’re partly an egoist and partly an altruist, then if you say that the altruistic component is on this kind of deliberation ladder then maybe you should go more with the egoistic part, until and unless you can find stability in your altruistic deliberations.
- Page 59
-
- [note::Basically... Go with your gut unless your moral uncertainty diminishes to acceptable levels?]

I’m going to argue that having less evidence in favor of a given intervention means that your credences about the effectiveness of that intervention are what I call “low resilience”.
- Page 60
-

And, I’m going to argue that this means that actually in many cases, we should prefer interventions with less evidential support, all else being equal. Hopefully, you’ll find that counterintuitive and interesting.
- Page 60
- interventions, information value,

We basically think it’s really likely that we’re going to move around a lot in response to new evidence. We’re just not willing to assert a credence that we think is just going to be false, or inaccurate once we gain a little bit more evidence.
- Page 64
-

Take this third case. You start to test the untested coin, so you perform a series of flips with the coin, and you start to see a pattern [Figure 3]. In a case like this, it looks like the coin in front of you is pretty heavily heads biased, or you at least start to quite rapidly increase your credence that it’s heads biased. So, your credence that it’s going to come up heads next time is much higher. Because you had less evidence before, this credence was much more fragile, so now you’ve seen a change. This would not happen if you got this sequence on the well-tested coin, because more evidence means that your credences are more resilient. If you saw a series of five head after performing a million trials, and it lands heads roughly half the time, this is just not going to make a huge difference to what you expect the next coin flip to be.
- Page 64
-

expected utility theory or expected value calculations might say that in cases where all else is equal, we should favor investing in interventions that have less evidence, rather than interventions that have more.
- Page 66
- evidence, cause prioritization, expected value,

The idea here is that in such cases, the concrete value is the same but, the information value for one of them is much higher, namely the one where you have a much lower resilience credence generating your expected value calculation
- Page 66
-

Explore, Exploit or Evade When we want to do good we can: Explore: Invest resources in intervention x for its information value • research, funding to gather data, career trials Exploit: Invest resources in intervention x for its concrete value • large project grants, career choices Evade: Do not invest in intervention x • investing elsewhere, delaying investment
- Page 67
-

(you don’t really want to be looking for information when cars are driving towards you, as the cost of not just taking action and getting out of the way is pretty high!).
- Page 67
-
- [note::Great analogy!]

So, when is exploring especially cost effective? Essentially, when there are three features. When there’s more uncertainty about the direct value of an intervention, so this means options that have high expected value, but low resilience. When there are high benefits of certainty about the direct value, so when we can basically repeatedly mine something for value. And when there are low information costs, so when information’s not too costly to obtain and the delay is low cost
- Page 67
-

The main difference between these is the reason for action.
- Page 67
-

“We should probably start explicitly including the value of information, and assessments of causes and interventions, rather than treating it as an afterthought to concrete value.”
- Page 68
-

Maybe you’re a bit like Carla. Maybe you’re very worried that we’re just screwing up the climate or that nuclear war is going to go terribly wrong. In which case, maybe you think we should just be directly intervening in those areas.
- Page 68
-

Effective altruism isn’t like a short-term intervention. So, in multi-generational projects, you expect the value of information to be higher, because people can essentially explore for longer and find optimal interventions.
- Page 68
- projects, information value, movements,

The Open Philanthropy Project suggests that “as the world becomes more interconnected, the magnitude and implications of the worst-case scenarios may be rising.”
- Page 73
-

temporal discounting, which means that we tend to give less weight to outcomes that are far in the future.
- Page 74
- forecasting, moral weight, temporal discounting, longtermism, cognitive bias,

Efforts to shape the long-term future could be extremely high in expected value
- Page 74
-

2. Objecting to the way that expected values are estimated in practice
- Page 77
- h4,

1. Objecting to the use of expected values to make decisions in principle
- Page 77
- h4,

Many attempts to do good fail, but the best are exceptional
- Page 6
-

Introduction to Effective Altruism
- Page 6
-

By remaining open to working on different causes, we’re able to change course to where we can make the biggest difference, without restricting ourselves too early.
- Page 8
-

The cause that you choose to work on is a big factor in how much good you can do. If you choose a cause where it’s not possible to help very many people, or where there just aren’t any good ways to solve the relevant problems, then you will significantly limit the amount of impact you can have.
- Page 8
-

useful.
- Page 8
-

US households spend around 2% of their budgets on personal insurance, on average. If we were to spend a comparable percentage of
- Page 9
-

Improving the long-term future
- Page 9
- h4,

Animal suffering
- Page 9
- h4,

Fighting extreme poverty
- Page 9
- h4,

Improvements to the scientific establishment, such as greater transparency and replication of results
- Page 10
-

Prevention of road traffic injuries
- Page 10
-

(highlight:: 1. Consider which problem you should focus on.
2. Consider the most effective way for you to address the problem)
- Page 10
-

US criminal justice reform
- Page 10
-

Other causes
- Page 10
-

Pledging to give
- Page 11
- h4,

Donating effectively
- Page 11
-

opportunity costs: buying one thing costs money that could be used to buy others
- Page 13
-

High overhead costs are only one possible failure mode for a charity. Consider again the Arctic explorer, trying to decide between a $200 parka and a $200 digital camera. Perhaps a parka only cost $100 to make and the manufacturer takes $100 profit, but the camera cost $200 to make and the manufacturer is selling it at cost. This speaks in favor of the moral qualities of the camera manufacturer, but given the choice the explorer should still buy the parka. The camera does something useless very efficiently, the parka does something vital inefficiently. A parka sold at cost would be best, but in its absence the explorer shouldn’t hesitate to choose the parka over the camera.
- Page 14
-

And likewise, there is only one best charity: the one that helps the most people the greatest amount per dollar. This is vague, and it is up to you to decide whether a charity that raises forty children’s marks by one letter grade for $100 helps people more or less than one that prevents one fatal case of tuberculosis per $100 or one that saves twenty acres of rainforest per $100. But you cannot abdicate the decision, or you risk ending up like the 11,000 people who accidentally decided that a pretty picture was worth more than a thousand people’s lives.
- Page 14
-

If a high-powered lawyer who makes $1,000 an hour chooses to take an hour off to help clean up litter on the beach, he’s wasted the opportunity to work overtime that day, make $1,000, donate to a charity that will hire a hundred poor people for $10/hour to clean up litter, and end up with a hundred times more litter removed
- Page 15
- impact,
- [note::EA in a nutshell - while valid, this kind of thinking can be incredibly toxic.]

The Roman historian Sallust said of Cato “He preferred to be good, rather than to seem so”. The lawyer who quits a high-powered law firm to work at a nonprofit organization certainly seems like a good person. But if we define “good” as helping people, then the lawyer who stays at his law firm but donates the profit to charity is taking Cato’s path of maximizing how much good he does, rather than how good he looks.
- Page 15
-
- [note::I love this phrasing - "Being good as opposed to seeming good"]

counterfactually
- Page 17
- belonging, community building, welcomeness,
- [note::Huh, I remember this word being very foreign to me when I first started becoming involved in EA. I should make a point to remember this feeling when talking to new people in the EA Philly community.]

When you see gold, take a moment to think about what you value. Many people won’t just value one particular thing. However, do think about what you care about and put that in place of the gold.
- Page 17
-
- [note::This seems like a useful exercise in terms of figuring out what top-level goals one should strive for.]

Elsewhere in this conference, there are going to be treasure maps and discussions of where the gold is. I’m not going to do that in this talk. I’m instead going to be focusing on the tools and techniques that we can use for locating gold, rather than trying to give my view of where it is directly.
- Page 18
-
- [note::I like this framing - EA is really all about finding out what and where the gold is ("gold" being what we altruistically value)]

So, on the left of the distribution that is not heavy-tailed, I can see that the gold is fairly evenly spread across a lot of different places. If we want to just get most of the gold, what is important is getting to as many different places as possible. Solar power is like this. Sure, some places get more sunlight than other places, but the amount of solar power you generate depends more on how many total solar panels you have, than on exactly where you place them. Over on the right, though, we have a distribution where you can see a lot of the area that spikes on the right-hand side. This just means that a lot of the gold, and if this is an actual stand-in for something that we value, a lot of what is valuable comes in this extreme of the distribution of things, which are just unusually good.
- Page 20
-

So just because somebody says, “Hey, this is gold,” doesn’t mean we should always take people’s word for it.
- Page 24
-

A nice property of literal gold is that when you dig it up, you’re pretty sure that you can recognise “yes I have gold”. We often have to deal with cases where we don’t have this. We don’t have the gold, so we have to carefully try to infer its existence, by using different tools. This fact is like the dark matter of value.
- Page 24
-
- [note::"Altruistic value is like dark matter - we have to infer it's existence by using different tools"]

You can still get gold, but it’s more work for each little bit, for each nugget that you’re getting out. This is the general phenomenon of ‘diminishing returns’ on work that you’re putting in. This concept comes up in a lot of different places, and so it is worth having an understanding of it.
- Page 25
- diminishing returns,

Running out of easy gold
- Page 25
-

That is great because we have taken those low hanging fruit. Or similarly, if in AI safety, writing the first book on superintelligence is a pretty big deal. Writing the 101st book on superintelligence is just not going to matter as much.
- Page 26
-

The first of these dimensions is scale. All else being equal, we would prefer to go somewhere where there is a lot of gold, rather than a little bit of gold.
- Page 26
-

Second: tractability. We’d like to go somewhere where we make more progress per unit work. Somewhere, where it’s nice and easy to dig the ground, rather than trying to get your gold out of a swamp.
- Page 27
-

And third is uncrowdedness. This has sometimes been called neglectedness. I think that term is a bit confusing. It’s a bit ambiguous because sometimes people use ‘neglectedness’ to mean that this is an area which we should allocate more resources to. What I mean here is that there aren’t many people looking at it. All else being equal, we would rather go to an area where people haven’t already picked up the nuggets of gold on the ground, than one where they have. And now the only gold remaining is quite hard to extract.
- Page 27
-
- [note::Nuanced yet important distinction between "neglectedness'" and "uncrowdedness]

The first term is measuring the amount of value you get for, say, solving an extra one percent of a solution.
- Page 28
-

It is an elasticity here, which is a technical term. It’s a pretty useful and general term (go look it up on Wikipedia, if you’re interested). Here, it is measuring, for a proportional increase in the amount of work that’s being done, what proportion of a solution that gives you.
- Page 28
-
- [note::#research-this]

The final term just cancels to one over the total amount of work being done. So that is very naturally a measure of uncrowdedness.
- Page 28
-

I’m not going to spend an hour helping a bee, even if nobody else is helping it and it would be pretty easy to help, because just the scale of it is pretty small. I don’t think we should work on perpetual motion machines, even though basically nobody is working on it and it would be really fantastic if we succeeded. Because it seems like it’s not tractable. And this [...] might give us a warning against actually working on climate change. Because at a global scale, that gets a lot of attention, as a problem.
- Page 29
- tractability, itn framework, importance, neglectedness,

Absolute and marginal priority
- Page 30
- 1action/research,

If we are having a conversation about what we as individuals or as small groups should do, I think it’s appropriate to use this notion of marginal priority, of how much do extra resources help. If we’re talking about what we collectively as a society or the world should do, I think it’s often correct to talk about this kind of notion of absolute priority and how much resources ought to be invested in it, total.
- Page 31
-
- [note::Important distinction]

And maybe we have some technologies which are destructive. So we can use dynamite and dynamite gets us loads of gold now, but it also blows up some gold, and now we never get that gold later. That could be pretty good if you are focusing just on trying to get gold in the short term. But it could be bad from this eventual gold perspective.
- Page 31
-

And so one of the major drivers of how much gold is eventually extracted is the order in which the technologies are developed, and the sequencing. If we discover the dynamite first, people are going to go and have fun with their dynamite and they’re going to destroy a lot of the gold. If we discover the drill first, then by the time dynamite comes along, people will go “Well, why would we use that? We have this fantastic drill.”
- Page 32
-

Comparative advantage
- Page 32
-
- [note::How can we, as a community, figure out how to effectively distribute our talents? Perhaps performing analytics on EA Hub cause area data?]

I have Harry, Hermione and Ron, and they have three tasks that they need to do to get some gold. They need to do some research, they need to mix some potions, and they need to do some wand work. Hermione is the best at everything, but she doesn’t have a time turner, so she can’t do everything. So we need to have some way of distributing the work. This is the idea of comparative advantage. Hermione has an absolute advantage on all of these tasks, but it would be a waste for her to go and work on the potions because Harry is not so bad at potions. And really, nobody else is at all good at doing the research in the library. So we should probably put her on this.
- Page 33
- coordination, comparative advantage,

This is a bit more speculative, but I think we can also apply this at the time level. We can ask ourselves, “What are we, today, the people of 2016, particularly well suited to do, versus people in the past and people in the future?” We can’t change what people in the past did. But we can make a comparison of what our comparative advantage is relative to people in the future. And if there is a challenge, if there were going to be some different possible challenges in the future that we need to meet, it makes sense that we should be working on the early ones. Because if challenges are coming in 2020, the people in 2025 just do not have a chance to work on that.
- Page 33
-

So, maybe it is the case that we can adapt one of these existing institutions for our purpose, which is trying to aggregate knowledge about what are the ways to go and do the most good. But maybe we want something a bit different, and maybe somebody in this room is going to do some work on coming up with valuable institutions for this. I actually think this is a really important problem. And it’s one that is going to become more important for us to deal with as a community, as the community grows.
- Page 35
-

Do you believe it because somebody else told you? Do you believe it because you have really thought this through carefully and worked it out for yourself? There is a blur between those. Often somebody tells you, and they give you some reasons. And you are thinking: “Oh, those reasons kind of check out,” but you haven’t deeply examined the argument yourself.
- Page 36
- deferring, favorite,
- [note::This absolutely applies to me. Though, it's hard to develop a well-informed opinion about something. That's why I'm so reluctant to do it.]

I’m shortening the chain of Chinese whispers, of passing this message along. And as things get passed along, it’s possible that mistakes enter or just something isn’t well grounded, and then it gets repeated. By going back and checking earlier sources in the chain, we can try to reduce that, and try to make ourselves more robustly confident in these statements.
- Page 37
-
- [note::"Shortening the chain of whispers" has risks associated with it (related to "value of information"?)]

Heavy-tailedness isn’t just a binary property; it’s a continuum
- Page 38
-
- [note::Not really sure what this means #research-this]

Heavy-tailed distributions
- Page 38
-

If people are pretty good at identifying the best opportunities, and they are uniformly seeking out and taking them, then the best things that are left might not be so much better
- Page 39
-

Ways to make money, maybe they actually start out distributed across a wide range. This is a log scale now, and it is meant to represent one of those heavy-tailed distributions, but then people who are losing money, say, “Well, this sucks,” and they stopped doing that thing. And they see other people who are doing activities which are making lots of money. And they think: “Yeah, I’m going to go do that.” And then you get more people going into that area, and then diminishing returns mean that you actually make less money than you used to, by doing stuff in that area.
- Page 39
-

Differential progress
- Page 42
-
- [note::#research-this]

Conclusion
- Page 44
-

crucial consideration
- Page 45
-
- [note::#research-this]

a crucial consideration is a consideration such that if it were taken into account it would overturn the conclusions we would otherwise reach about how we should direct our efforts, or an idea or argument that might possibly reveal the need not just for some minor course adjustment in our practical endeavors but a major change of direction or priority.
- Page 45
- crucial considerations,

deliberation ladder, which would be a sequence of crucial considerations, regarding the same high-level subgoal, where the considerations hold in opposing directions.
- Page 46
-

So a crucial consideration component will be an argument, idea or datum which, while not on its own amounting to a crucial consideration, seems to have a substantial probability of maybe being able to serve a central role within a crucial consideration.
- Page 46
-

technological completion conjecture — that in the fullness of time, unless civilization collapses, all possible general useful technologies will be developed —, these dangers will have to be confronted, and all our choice really concerns is the sequence in which we confront them.
- Page 48
-

Whereas with utilitarianism, the utilitarian preference extends far and wide beyond our familiar environment, including into the cosmic commons and billions of years into the future and super advanced civilizations: what they do matters from the utilitarian perspective, and matters a lot. Most of what the utilitarian preference cares about is stuff that we have no familiarity with.
- Page 49
-

if one tries to think about how to apply utilitarianism to a world that has a finite probability of being infinite, one will run into difficulties in terms of how to measure different infinite magnitudes and still seeing how we could possibly make any difference to it.
- Page 49
-

I think that these deliberation ladders are particularly likely to turn up when one is trying to be a thoroughgoing utilitarian and one really takes the big-picture question seriously.
- Page 49
-

The third possible reason here is that one might think that we are kind of close, not super close, but close to some pivot point in history. That means that we might have special opportunities to influence the long-term future now. And we’re still far enough away from this: it’s not obvious what we should do to have the maximally beneficial impact on the future. But still close enough that we can maybe begi
- Page 49
-

This could affect the personal domain as well. It’s just like with an ordinary person’s typical utility function: they probably don’t place a million times more value on living for a billion years than living for a hundred years, or a thousand times more value on raising a thousand children than on raising one child. So even though the future still exists, it just doesn’t weigh as heavily in a normal human utility function as it does for utilitarians.
- Page 50
-

50 P R I N C I P L E S to perceive some contours of the apparatus that will shape the future.
- Page 50
-

We do something analogous to that in other domains. Like a typical traditional public policy, social welfare economists might think that you need to maximize some social welfare function which might take a form like this.
- Page 51
-

If we plot capacity, which could be level of economic development and technological sophistication, stuff like that, on one axis and time on the other, my view is that the human condition is a kind of metastable region on this capability axis. You might fluctuate inside for a while, but the longer the time scale you’re considering, the greater the chance that you will exit that region in either the downwards direction and go extinct
- Page 51
-

Maxipok principle that Beckstead also mentioned: Maximize the probability of an OK outcome.
- Page 52
-

principle of differential technological development suggests that we should retard the development of dangerous and harmful technologies—once the raise existential risk, that is— and accelerate technologies that reduce existential risks.
- Page 54
-

The point here is that to have the best possible condition; we need super advanced technology: to be able to access the cosmic commons, to be able to cure all the diseases that plague us, etc. I think to have the best possible world, you’ll also need a huge amount of insight and wisdom, and a large amount of coordination so as to avoid using high technology to wage war against one another, and so forth.
- Page 54
-
- [note::This feels almost greedy (i.e. why can't we stay in our current situation?), but the reality is that we can't really hover - human progress is unlikely to stop because things like impending climate change and our current economic system will it to continue.]

However, we might feel it’s very difficult to get strong reasons for knowing whether more people in this room is better, or whether there is some inverse relationship. A good signpost would strike a reasonable compromise between being visible from afar and also being such that we can have strong reason to be sure of its sign.
- Page 55
-
- [note::I don't really understand the "being visible from afar" part]

Nietzschean Umwertung aller Werte — the revaluation of all values
- Page 57
-
- [note::Huh?]

To just pick an example: insects. If you are a classical utilitarian, this consideration arises within the more mundane—we’re setting aside the cosmological commons and just thinking about here on Earth. If insects are sentient then maybe the amount of sentience in insects is very large because there are very,
- Page 57
-

Take into account that expected value changes are probably smaller than they appear. If you are a utilitarian, let’s say you think of this new argument that has this radical implication for what you should be doing, the first instinct might be to radically change your expected utility of different practical policies in light of this new insight. But maybe when you reflect on the fact that there are new crucial considerations being discovered every once in awhile, maybe you should still change List of some areas with candidate remaining CCs or CCCs • Counterfactual trade • Simulation Stuff • Infinite paralysis • Pascalian muggings • Different kinds of aggregative ethics (total, average, negative) • Information hazards • Aliens • Baby universes • Other kinds of moral uncertainty • Other game theory stuff ` • Pessimistic metainduction; epistemic humility; anthropics • Insects, subroutine
- Page 58
-
- [note::Hmmm... this is good to keep in mind. "If a new insight has the tendency to radically change your perception in one direction, then it's not impossible that another new insight could radically change your perception in the opposite direction, and so your overall perception might not change as much as you think it will."]

very many of them. So that maybe the effect of our policies on insect well-being mind trump the effect of our policies on human well-being or animals in factories and stuff like that. I’m not saying it does, but it’s a question that is non-obvious and that could have a big impact.
- Page 58
-

59 P R I N C I P L E S your expected value, but not as much as it seems you should the first time.
- Page 59
-

Parliamentary Model
- Page 59
-

This is the idea that if you are unsure as to which moral theory is true, then you should assign probabilities to different moral theories and imagine that there were a parliament where each moral theory got to send delegates to that parliament in proportion to their probability.
- Page 59
-

The idea is that, other things equal, the more probability a moral theory has, the greater its say in determining your actions, but there might also be these trades between different moral theories
- Page 59
-

So if you’re partly an egoist and partly an altruist, then if you say that the altruistic component is on this kind of deliberation ladder then maybe you should go more with the egoistic part, until and unless you can find stability in your altruistic deliberations.
- Page 59
-
- [note::Basically... Go with your gut unless your moral uncertainty diminishes to acceptable levels?]

I’m going to argue that having less evidence in favor of a given intervention means that your credences about the effectiveness of that intervention are what I call “low resilience”.
- Page 60
-

And, I’m going to argue that this means that actually in many cases, we should prefer interventions with less evidential support, all else being equal. Hopefully, you’ll find that counterintuitive and interesting.
- Page 60
- interventions, information value,

We basically think it’s really likely that we’re going to move around a lot in response to new evidence. We’re just not willing to assert a credence that we think is just going to be false, or inaccurate once we gain a little bit more evidence.
- Page 64
-

Take this third case. You start to test the untested coin, so you perform a series of flips with the coin, and you start to see a pattern [Figure 3]. In a case like this, it looks like the coin in front of you is pretty heavily heads biased, or you at least start to quite rapidly increase your credence that it’s heads biased. So, your credence that it’s going to come up heads next time is much higher. Because you had less evidence before, this credence was much more fragile, so now you’ve seen a change. This would not happen if you got this sequence on the well-tested coin, because more evidence means that your credences are more resilient. If you saw a series of five head after performing a million trials, and it lands heads roughly half the time, this is just not going to make a huge difference to what you expect the next coin flip to be.
- Page 64
-

expected utility theory or expected value calculations might say that in cases where all else is equal, we should favor investing in interventions that have less evidence, rather than interventions that have more.
- Page 66
- evidence, cause prioritization, expected value,

The idea here is that in such cases, the concrete value is the same but, the information value for one of them is much higher, namely the one where you have a much lower resilience credence generating your expected value calculation
- Page 66
-

Explore, Exploit or Evade When we want to do good we can: Explore: Invest resources in intervention x for its information value • research, funding to gather data, career trials Exploit: Invest resources in intervention x for its concrete value • large project grants, career choices Evade: Do not invest in intervention x • investing elsewhere, delaying investment
- Page 67
-

(you don’t really want to be looking for information when cars are driving towards you, as the cost of not just taking action and getting out of the way is pretty high!).
- Page 67
-
- [note::Great analogy!]

So, when is exploring especially cost effective? Essentially, when there are three features. When there’s more uncertainty about the direct value of an intervention, so this means options that have high expected value, but low resilience. When there are high benefits of certainty about the direct value, so when we can basically repeatedly mine something for value. And when there are low information costs, so when information’s not too costly to obtain and the delay is low cost
- Page 67
-

The main difference between these is the reason for action.
- Page 67
-

“We should probably start explicitly including the value of information, and assessments of causes and interventions, rather than treating it as an afterthought to concrete value.”
- Page 68
-

Maybe you’re a bit like Carla. Maybe you’re very worried that we’re just screwing up the climate or that nuclear war is going to go terribly wrong. In which case, maybe you think we should just be directly intervening in those areas.
- Page 68
-

Effective altruism isn’t like a short-term intervention. So, in multi-generational projects, you expect the value of information to be higher, because people can essentially explore for longer and find optimal interventions.
- Page 68
- projects, information value, movements,

The Open Philanthropy Project suggests that “as the world becomes more interconnected, the magnitude and implications of the worst-case scenarios may be rising.”
- Page 73
-

temporal discounting, which means that we tend to give less weight to outcomes that are far in the future.
- Page 74
- forecasting, moral weight, temporal discounting, longtermism, cognitive bias,

Efforts to shape the long-term future could be extremely high in expected value
- Page 74
-

2. Objecting to the way that expected values are estimated in practice
- Page 77
- h4,

1. Objecting to the use of expected values to make decisions in principle
- Page 77
- h4,