How to Think Clearly (9/11): Explore Early, Exploit Late, Imagine the Failure First
March's exploration-exploitation trade-off, a small bandit simulation, and Klein's premortem: how to spend the cost of learning.
Where this sits: stage 5 of 6, Direction. Once estimates exist, the question is where to point effort: how to price time (post 8), how to split effort between finding out and using what is known (this post), and how to tell a small peak from a high one (post 10).
Where this post sits in the series: stage 5 of 6, Direction.
The estimation post gave tools for producing estimates. Two questions come after: how much of your effort to spend finding out versus using what you already know, and how to test a plan before committing. Both are about the cost of learning, which is paid at different times.
Two activities that compete
In 1991 James March described the trade-off in organizations. Exploration includes “search, variation, risk taking, experimentation, play, flexibility, discovery, innovation.” Exploitation includes “refinement, choice, production, efficiency, selection, implementation, execution.” The essence of exploitation, he wrote, is refining existing competences, and “its returns are positive, proximate, and predictable.” Exploration’s returns are “uncertain, distant, and often negative.”
That asymmetry explains the bias. Exploitation pays now and visibly; exploration pays later or not at all. March’s summary of the risk is that systems which only exploit “are likely to find themselves trapped in suboptimal stable equilibria,” and his abstract warns that adaptive processes refining exploitation faster than exploration become “effective in the short run but self-destructive in the long run.”
A simulation
Choosing among options of unknown quality is the multi-armed bandit problem, which Herbert Robbins popularised in 1952. I ran a small version for this post: ten options with hidden average rewards, 1,000 choices, and three strategies, each averaged over 1,500 runs.
Simulation by the author. The setup (ten options, Gaussian rewards, the exploration schedule) is my assumption; the numbers illustrate a pattern and are not a general result.
Over the whole run the average reward was 1.04 for never exploring, 1.31 for exploring a fixed 10% of the time, and 1.40 for exploring heavily at first and tapering to zero by choice 400. In the first 100 choices the picture is different: never exploring was slightly ahead of the tapering strategy, 0.97 against 0.92. Exploration costs something up front, and the payoff comes only if there is time left to use what was learned.
That is the whole argument for exploring early, and also the reason it should end. It is a statement about horizon: with many choices left, information is worth buying; with few, it is not. The right mix depends on how much time remains, which is why it is usually stated for stages of a career or a project, not as a fixed ratio.
Imagine the failure first
Exploitation needs a plan, and plans are optimistic, as the estimation post showed. Gary Klein’s premortem, described in Harvard Business Review in September 2007, is a procedure against that. He starts from research conducted in 1989 by Deborah Mitchell, Jay Russo and Nancy Pennington which, as Klein reports it, found that “prospective hindsight,” imagining that an event has already occurred, “increases the ability to correctly identify reasons for future outcomes by 30%.” I did not read the 1989 study.
The procedure is short. After the team has been briefed on the plan, the leader announces that the project has failed spectacularly. Everyone then writes down, independently, every reason they can think of, “especially the kinds of things they ordinarily wouldn’t mention as potential problems, for fear of being impolitic.”
Klein’s contrast is with an ordinary critique session. Asking “what might go wrong” invites a defence of the plan. Asking “what did go wrong” assumes the failure and asks for an explanation, which is an easier task and a safer one to answer honestly.
In machine learning
Both ideas are standard machinery in the field. In the bandit chapter of Sutton and Barto’s textbook Reinforcement Learning: An Introduction, picking the action with the highest current estimate is called exploiting, and picking another is exploring, “because this enables you to improve your estimate of the nongreedy action’s value.” Exploitation, they write, is right for maximizing expected reward on the one step, “but exploration may produce the greater total reward in the long run.” Their simplest remedy is the ε-greedy rule: act greedily most of the time and, with a small probability ε, pick an action at random. The strategies in the simulation above are of this family, with a fixed ε of 10% for “exploring a fixed share of the time” and an ε that decays for “exploring heavily at first and tapering”. The textbook also warns that its guarantees for ε-greedy are asymptotic and “say little about the practical effectiveness of the method”, which is the horizon point made above.
The premortem has a counterpart too. In a 2022 paper, Ethan Perez and colleagues note that language models often cannot be deployed because of potential harms that are hard to predict, and that earlier work found them with human-written test cases, which is expensive. They used another language model to generate test cases and found tens of thousands of offensive replies in a 280-billion-parameter chatbot, while calling the approach one promising tool among many needed. The logic matches the premortem, to find the failures before their cost is paid, but the method differs: a premortem imagines the failure, while red teaming searches for it by running the system. That comparison is this post’s own.
How they fit
The two ideas answer different questions. Exploration is how you find what to commit to; the premortem is how you test what you committed to before the cost is paid. Both spend a little now to avoid a larger loss later. And both fight the same pull, the pull of what pays visibly and soon: the exploit-now bias in March’s analysis, and the optimistic plan in the premortem. This pairing is my own reading, not a claim from either source.
The next post asks how to tell a small peak from a high one, and how to budget the descent from it. The series ends with the newest source of judgement in the loop, software that can think, and the question of which part of the decision remains yours.
Series: How to Think Clearly. 1. Where Thinking Breaks and What to Put There. 2. A Thought Has a Machine Behind It. 3. Catch the Error While It Happens. 4. Make the Right Thing the Easy Thing. 5. The Map Is Not the Territory. 6. Three Tools for Estimating the Unknown. 7. Averages Lie. 8. Price Your Own Hour. 9. Explore Early, Exploit Late, Imagine the Failure First. 10. The Hill You Are Standing On. 11. Let AI Think, Keep the Decision.