(highlight:: In short, our top recommendations are to:
Open with a linked summary of key takeaways. [more]
Throughout a document, indicate which considerations are most important to your key takeaways. [more]
Throughout a document, indicate how confident you are in major claims, and what support you have for them. [more])
- No location available
-

(highlight:: When reading an analysis — e.g. a scientific paper, or some other collection of arguments and evidence for some conclusions — we want to know: “How should I update my view in response to this?” In particular, we want to know things like:
Has the author presented a fair or biased presentation of evidence and arguments on this topic?
How much expertise does the author have in this area?
How trustworthy is the author in general? What are their biases and conflicts of interest?
What was the research process that led to this analysis? What shortcuts were taken?
What rough level of confidence does the author have in each of their substantive claims?
What support does the author think they have for each of their substantive claims?
What does the author think are the most important takeaways, and what could change the author’s mind about those takeaways?
If the analysis includes some data analysis, how were the data collected, which analyses were done, and can I access the data myself?)
- No location available
-

Consider GiveWell’s review of Against Malaria Foundation (AMF. This review…
…includes a summary of the most important points of the review, each linked to a longer section that elaborates those points and the evidence for them in some detail.
…provides detailed responses to major questions that bear on the likely cost-effectiveness of marginal donations to AMF, e.g. “Are LLINs targeted at people who do not already have them?”, “Do LLINs reach intended destinations?”, “Is there room for more funding?”, and “How generally effective is AMF as an organization?”
…provides a summary of the research process GiveWell used to evaluate AMF’s cost-effectiveness.
…provides an endnote, link to another section or page, or other indication of reasoning/sources for nearly every substantive claim. There are 125 endnotes, and in general, the endnote provides the support for the corresponding claim, e.g. a quote from a scientific paper, or a link to a series of calculations in a spreadsheet, or a quote from a written summary of an interview with an expert. (There are some claims that do not have such support, but these still tend to clearly signal what the basis for the claim is; e.g. “Given that countries and other funders have some discretion over how funds will be used, it is likely that some portion of AMF’s funding has displaced other funding into other malaria interventions and into other uses.”)
…provides a comprehensive table of sources, including archived copies of most sources in case some of the original links break at some point.
…includes a list of remaining open questions about AMF’s likely cost-effectiveness, plus comments throughout the report on which claims about AMF GiveWell is more or less confident in, and why.
…links to a separate summary of the scientific evidence for the effectiveness of the intervention AMF performs, namely the mass distribution of long-lasting insecticide-treated nets (LLINs), which itself exhibits all the features listed above.
…plus much more)
- No location available
-

Which arguments or pieces of evidence are most critical for your key takeaways? Ideally, this should be made clear early in the document, or at least early in the section discussing each key takeaway.
- No location available
-

For most substantive claims, or at least for “major” claims that are especially critical for your conclusions, try to give some indication of how confident you are in each claim, and what support you think you have for it.
- No location available
-

Confidence in a claim can be expressed roughly using words such as “plausible,” “likely,” “unlikely,” “very likely,” and so on. When it’s worth the effort, in some cases you might want to express your confidence as a probability or a confidence interval, in part because terms like “plausible” can be interpreted differently by different readers.
- No location available
-

“It is widely believed, and seems likely, that regular, high-quality sleep is important for personal performance and well-being, as well as for public safety and other important outcomes” (source). This claim was not a major focus of the report, so I simply said “seems likely,” to indicate that I think the probability of my statement being true is >50%, while also indicating that I haven’t investigated the evidence in detail and haven’t tried to acquire a more precise probabilistic estimate.
- No location available
-

Given limited resources, you cannot systematically and carefully examine every argument and piece of evidence relevant to every claim in your analysis. Nor can you explain in detail what kind(s) of support you think you have for every claim you make. Nevertheless, you can quickly give the reader some indication of what kind(s) of support you have for different claims, and you can explain in relatively more detail the kind(s) of support you think you have for some key claims.
- No location available
-

(highlight:: Here are some different kinds of support you might have for a claim:
another detailed analysis you wrote
careful examination of one or more studies you feel qualified to assess
careful examination of one or more studies you feel only weakly able to assess
shallow skimming of one or more studies you feel qualified to assess
shallow skimming of one or more studies you feel only weakly able to assess
verifiable facts you can easily provide sources for
verifiable facts you can’t easily provide sources for
expert opinion you feel comfortable assessing
expert opinion you can’t easily assess
a vague impression you have based on reading various sources, or talking to various experts, or something else
a general intuition you have about how the world works
a simple argument that seems robust to you
a simple argument that seems questionable to you
a complex argument that nevertheless seems strong to you
a complex argument that seems questionable to you
the claim seems to follow logically from other supported claims plus general background knowledge
a source you can’t remember, except that you remember thinking at the time it was a trustworthy source, and you think it would be easy to verify the claim if one tried[4]In such a case, you might say something like: “We do not recall our source for this information but believe it would be straightforward to verify.”
a combination of any of the above)
- No location available
-

(highlight:: Here’s the first example:
“As stated above, my view is based on a large number of undocumented conversations, such that I don’t think it is realistic to aim for being highly convincing in this post. Instead, I have attempted to lay out the general structure of the inputs into my thinking. For further clarification, I will now briefly go over which parts of my argument I believe are well-supported and/or should be uncontroversial, vs. which parts rely crucially on information I haven’t been able to fully share…” [source])
- No location available
-

In some cases, you can’t provide much of the reasoning for your view, and it’s most transparent to simply say so.
- No location available
-

When citing some support for a claim, provide a page number if possible. Even better if you can directly quote the most relevant passage, so the reader doesn’t need to track down the source to get a sense of what kind of support for the claim the source provides.
- No location available
-

E.g. see the many long quotes in the footnotes of my report on consciousness and moral patienthood, or the dozens of quotes in the footnotes of GiveWell intervention reports.
- No location available
-

For many investigations, interviews with domain experts will be a key source of information alongside written materials. Hence when possible, it will help improve the reasoning transparency of a report if those conversations (or at least the most important ones can be made available to the reader, either as a transcript or a summary.
But in many cases this is too time-costly to be worth doing, and in many cases a domain expert will only be willing to speak frankly with you anonymously, or will only be willing to be quoted/cited on particular points.)
- No location available
-

(highlight:: In short, our top recommendations are to:
Open with a linked summary of key takeaways. [more]
Throughout a document, indicate which considerations are most important to your key takeaways. [more]
Throughout a document, indicate how confident you are in major claims, and what support you have for them. [more])
- No location available
-

(highlight:: When reading an analysis — e.g. a scientific paper, or some other collection of arguments and evidence for some conclusions — we want to know: “How should I update my view in response to this?” In particular, we want to know things like:
Has the author presented a fair or biased presentation of evidence and arguments on this topic?
How much expertise does the author have in this area?
How trustworthy is the author in general? What are their biases and conflicts of interest?
What was the research process that led to this analysis? What shortcuts were taken?
What rough level of confidence does the author have in each of their substantive claims?
What support does the author think they have for each of their substantive claims?
What does the author think are the most important takeaways, and what could change the author’s mind about those takeaways?
If the analysis includes some data analysis, how were the data collected, which analyses were done, and can I access the data myself?)
- No location available
-

Consider GiveWell’s review of Against Malaria Foundation (AMF. This review…
…includes a summary of the most important points of the review, each linked to a longer section that elaborates those points and the evidence for them in some detail.
…provides detailed responses to major questions that bear on the likely cost-effectiveness of marginal donations to AMF, e.g. “Are LLINs targeted at people who do not already have them?”, “Do LLINs reach intended destinations?”, “Is there room for more funding?”, and “How generally effective is AMF as an organization?”
…provides a summary of the research process GiveWell used to evaluate AMF’s cost-effectiveness.
…provides an endnote, link to another section or page, or other indication of reasoning/sources for nearly every substantive claim. There are 125 endnotes, and in general, the endnote provides the support for the corresponding claim, e.g. a quote from a scientific paper, or a link to a series of calculations in a spreadsheet, or a quote from a written summary of an interview with an expert. (There are some claims that do not have such support, but these still tend to clearly signal what the basis for the claim is; e.g. “Given that countries and other funders have some discretion over how funds will be used, it is likely that some portion of AMF’s funding has displaced other funding into other malaria interventions and into other uses.”)
…provides a comprehensive table of sources, including archived copies of most sources in case some of the original links break at some point.
…includes a list of remaining open questions about AMF’s likely cost-effectiveness, plus comments throughout the report on which claims about AMF GiveWell is more or less confident in, and why.
…links to a separate summary of the scientific evidence for the effectiveness of the intervention AMF performs, namely the mass distribution of long-lasting insecticide-treated nets (LLINs), which itself exhibits all the features listed above.
…plus much more)
- No location available
-

Which arguments or pieces of evidence are most critical for your key takeaways? Ideally, this should be made clear early in the document, or at least early in the section discussing each key takeaway.
- No location available
-

For most substantive claims, or at least for “major” claims that are especially critical for your conclusions, try to give some indication of how confident you are in each claim, and what support you think you have for it.
- No location available
-

Confidence in a claim can be expressed roughly using words such as “plausible,” “likely,” “unlikely,” “very likely,” and so on. When it’s worth the effort, in some cases you might want to express your confidence as a probability or a confidence interval, in part because terms like “plausible” can be interpreted differently by different readers.
- No location available
-

“It is widely believed, and seems likely, that regular, high-quality sleep is important for personal performance and well-being, as well as for public safety and other important outcomes” (source). This claim was not a major focus of the report, so I simply said “seems likely,” to indicate that I think the probability of my statement being true is >50%, while also indicating that I haven’t investigated the evidence in detail and haven’t tried to acquire a more precise probabilistic estimate.
- No location available
-

Given limited resources, you cannot systematically and carefully examine every argument and piece of evidence relevant to every claim in your analysis. Nor can you explain in detail what kind(s) of support you think you have for every claim you make. Nevertheless, you can quickly give the reader some indication of what kind(s) of support you have for different claims, and you can explain in relatively more detail the kind(s) of support you think you have for some key claims.
- No location available
-

(highlight:: Here are some different kinds of support you might have for a claim:
another detailed analysis you wrote
careful examination of one or more studies you feel qualified to assess
careful examination of one or more studies you feel only weakly able to assess
shallow skimming of one or more studies you feel qualified to assess
shallow skimming of one or more studies you feel only weakly able to assess
verifiable facts you can easily provide sources for
verifiable facts you can’t easily provide sources for
expert opinion you feel comfortable assessing
expert opinion you can’t easily assess
a vague impression you have based on reading various sources, or talking to various experts, or something else
a general intuition you have about how the world works
a simple argument that seems robust to you
a simple argument that seems questionable to you
a complex argument that nevertheless seems strong to you
a complex argument that seems questionable to you
the claim seems to follow logically from other supported claims plus general background knowledge
a source you can’t remember, except that you remember thinking at the time it was a trustworthy source, and you think it would be easy to verify the claim if one tried[4]In such a case, you might say something like: “We do not recall our source for this information but believe it would be straightforward to verify.”
a combination of any of the above)
- No location available
-

(highlight:: Here’s the first example:
“As stated above, my view is based on a large number of undocumented conversations, such that I don’t think it is realistic to aim for being highly convincing in this post. Instead, I have attempted to lay out the general structure of the inputs into my thinking. For further clarification, I will now briefly go over which parts of my argument I believe are well-supported and/or should be uncontroversial, vs. which parts rely crucially on information I haven’t been able to fully share…” [source])
- No location available
-

In some cases, you can’t provide much of the reasoning for your view, and it’s most transparent to simply say so.
- No location available
-

When citing some support for a claim, provide a page number if possible. Even better if you can directly quote the most relevant passage, so the reader doesn’t need to track down the source to get a sense of what kind of support for the claim the source provides.
- No location available
-

E.g. see the many long quotes in the footnotes of my report on consciousness and moral patienthood, or the dozens of quotes in the footnotes of GiveWell intervention reports.
- No location available
-

For many investigations, interviews with domain experts will be a key source of information alongside written materials. Hence when possible, it will help improve the reasoning transparency of a report if those conversations (or at least the most important ones can be made available to the reader, either as a transcript or a summary.
But in many cases this is too time-costly to be worth doing, and in many cases a domain expert will only be willing to speak frankly with you anonymously, or will only be willing to be quoted/cited on particular points.)
- No location available
-