McNamara Fallacy

Notes

Claim

The flawed assumption that what can't be measured isn't important.

Explanation

Usually it's not that we actively and consciously dismiss all other metrics as not important, rather that we subconsciously treat only what's already measured as important. It is similar to Availability bias inflates perceived likelihood of easily recalled events, what is already measured is easier to recall which gives it a false pretense of importance. Additionally, we detest Ambiguity is the absence of clarity in values or facts, so anything that can't be simplified is too cognitive taxing, so we prefer to treat it as not important.

This can also easily lead to Goodhart's Law because laziness can push us towards finding an easy proxy to an abstract (yet important) goal, which we will treat as the goal itself, thus missing the actual target and perhaps even causing unwanted consequences Shallow copies mistake partial representations for complete truth. This essentially means that we also fall into the Focusing on easy metrics blinds us to what matters, where we cling to what's easy to measure versus what is actually important.

Sometimes we notice that we fall for this fallacy when it's clear that the We draw conclusions only from what survived is also relevant. When the metrics we use don't explain much in terms of success rates, i.e those who "score high" fail and those who "score low" succeed.

Why it matters

Usually this fallacy happens when we focus on numeric metrics since they are the easiest to use, which causes qualitative metrics to be discarded as not important Self-knowledge can be driven by data. However, much of expertise is based on knowledge that's difficult to quantify or even describe, but that doesn't mean that it's not important. It gets worse if we perceive what's measured not only as the most important metrics, but also that other metrics doesn't exist, i.e that these metrics are a pure representation of the information.

A way to reduce the chances of falling for this fallacy is to Generating multiple alternatives simultaneously improves outcomes, to use a variety of metrics which means we won't settle for the first who come to mind, rather force us to create more metrics which would hopefully will be a better representation of all that's important to measure.

Second, we must find ways of incorporating qualitative measures. While they are hard to quantify and reduce to something easier to digest and compare, they often contain valuable Feedback is information that enables behavioral improvement.

Examples

Supporters

Opposers

Open questions

Visual

Overview

🔼Topic:: Selective Evidence Origin:: The Curiosity Chronicle by Sahil Bloom 🔗Link::