adulthood as a policy network

Imagine you want to make decisions in the world. One way is to learn how the world works, consider possible approaches, and think through the right thing to do in every case. You could use an uncertainty-aware planning algorithm that searches over action sequences, allocating computation as appropriate until it gets to the point where it's confident it knows the right decision considering all factors. As you gain experience in the world, you'll drive down uncertainty on some aspects but also discover whole new factors to take into account.

Or you could use a policy network that does a fixed-time feedforward computation to output a decision.

In the real world, people often discuss what to do. Should we hire this person? Should we devote time to building good software or to novel research or to selling our work? Should a piece of code assume one sort of user or another sort of user? How should I structure my work on this project? etc etc.

Many of these questions are impossible to answer. They involve too many factors; you can never be confident you have the right answer.

And yet you need to say something. When advising someone you have to give them advice. In meetings you have to have a take. What should we do? Why? With what evidence? Society rewards confidence, which is extremely frustrating because if you're at all rational you realize it's impossible to confidently know the right thing to do in many of these situations.

The answer is to retreat from act utilitarianism (doing the right thing in every case) to rule utilitarianism (following rules that generally do the right thing). This is analogous to retreating from Bayesianism (finding the right estimate for this particular data) to frequentism (finding rules that generally give good estimates).

The thing that drives senior decision-makers is not necessarily seeing deeper or being more right, but having trained their policy networks to give generally good responses. They've found rules that are generally useful. They will confidently tell you what you should do in all sorts of situations where, in fact, it's really unknowable what you should do. But they'll have a confident answer that's part of a consistent philosophy, can be justified, is usually at least reasonable, and helps you move forward.

Of course this totally narrows their capabilities because they no longer try to do the impossible. They have a process and they execute the process, they don't look for cheat codes.

See also growing up means becoming wrong, process is frequentist