How to Think Clearly (5/11): The Map Is Not the Territory
Four ways a compressed picture misleads: the map, the metric, the fence and the unseen cost, with one question that covers them all.
Where this sits: stage 3 of 6, Belief. The first two stages dealt with the machine and its setting. This one deals with what the machine holds as true: every belief is a compressed picture of the world, and the safeguard is to check the picture against the territory.
Where this post sits in the series: stage 3 of 6, Belief.
The previous post argued for setting defaults and triggers in advance. That raises the question of which ones. The answer depends on a picture of the world, and every picture is a compression. This post is about four ways a compressed picture misleads the person holding it.
Figure 1. Four ways a compressed picture misleads, and the one question that checks all of them.
The map
Alfred Korzybski presented the phrase in 1931 at a meeting of the American Association for the Advancement of Science, and it was reprinted in Science and Sanity (1933). The sentence is longer than its slogan: “A map is not the territory it represents, but, if correct, it has a similar structure to the territory, which accounts for its usefulness.”
The second half matters as much as the first. A map that copied everything would be useless, and a map with nothing in common with the land would be wrong. A good map is usable because it omits things. The danger is forgetting that the omission happened.
The measure
A metric is a map of something you care about. Charles Goodhart’s 1975 statement, about British monetary policy, was: “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” The familiar version, “when a measure becomes a target, it ceases to be a good measure,” is Marilyn Strathern’s 1997 paraphrase, drawing on an earlier account by Keith Hoskin. I could not read Goodhart’s original paper, so both wordings come from a secondary source.
The mechanism, in my words: a measure works because it correlates with the thing behind it. Once people are rewarded for the number, they find the cheapest way to raise the number, and the cheap ways often leave the thing behind it untouched. The map gets improved while the land stays the same.
The fence
Some maps are drawn by others, long ago, and what is missing from your copy may be a reason. In The Thing (1929), in the chapter “The Drift from Domesticity,” G. K. Chesterton put it as a fence across a road. The modern reformer says, “I don’t see the use of this; let us clear it away.” The wiser reformer answers: “If you don’t see the use of it, I certainly won’t let you clear it away. Go away and think. Then, when you can come back and tell me that you do see the use of it, I may allow you to destroy it.”
His reason is plain: the fence “did not grow there,” and “some person had some reason for thinking it would be a good thing for somebody.” The rule is not “never remove fences.” It is “removal needs an explanation of why the fence was built.” If you cannot explain the fence, the gap is in your map.
The unseen
The last case is a missing entry on every map: what you gave up. In 1850 Frédéric Bastiat told the story of a broken window. The glazier is paid six francs and everyone sees the gain. But “it is not seen that as our shopkeeper has spent six francs upon one thing, he cannot spend them upon another.” He might have “replaced his old shoes, or added another book to his library.”
Economists call this opportunity cost. Bastiat’s framing is better for thinking: the chosen option is vivid and the foregone one is invisible, so the map is biased toward whatever happened.
In machine learning: a measure with a gradient
Training a language model with human feedback builds the map-and-measure problem into the pipeline. A reward model is trained to predict human preferences, and the language model is then optimized against it. In Gao, Schulman and Hilton’s 2022 study, the authors point out that the reward model is an imperfect proxy, so optimizing its value too much can hurt ground-truth performance, in accordance with Goodhart’s law. To measure it, they used a synthetic setup in which a fixed “gold-standard” reward model plays the role of humans and provides the labels for a proxy. They then tracked the gold score as optimization against the proxy increased, using either reinforcement learning or best-of-n sampling. The relationship took a different functional form for each method, and its coefficients scaled smoothly with the number of reward model parameters.
In this post’s terms, the proxy score is the map and the gold standard is the territory, and the optimizer is a very efficient search for the cheapest way to raise the number. One caution on the evidence: the gold standard in that study is itself a model, so it measures the gap between a proxy and a stronger judge, not the gap to real human judgement. The practical reading of the check below is this post’s own: keep a measurement that the optimizer cannot touch.
One check for all four
All four are the same error at different points: treating a compressed picture as the thing itself. One question covers them:
What does this picture leave out, and who decided it should?
For a map, the answer is the omitted terrain. For a metric, the thing behind the number. For a fence, the builder’s reason. For a choice, the road not taken. The next post gives three tools for the case where the missing piece cannot be looked up and has to be estimated.
Series: How to Think Clearly. 1. Where Thinking Breaks and What to Put There. 2. A Thought Has a Machine Behind It. 3. Catch the Error While It Happens. 4. Make the Right Thing the Easy Thing. 5. The Map Is Not the Territory. 6. Three Tools for Estimating the Unknown. 7. Averages Lie. 8. Price Your Own Hour. 9. Explore Early, Exploit Late, Imagine the Failure First. 10. The Hill You Are Standing On. 11. Let AI Think, Keep the Decision.