instrumental goal

To achieve final goals, we have to break them down into a hierarchy of instrumental goals, and then get to work on achieving those. And for sufficiently big final goals, like 'optimize human utility' or 'cure cancer' or 'ensure world peace', you're going to need a lot of instrumental goals and those goals themselves are going to be pretty daunting.

We build mechanisms to optimize instrumental goals. Corporations have optimizing shareholder value as an top-level instrumental goal (well, it is a final goal from the corporation's point of view, but an instrumental goal from the broader perspective of human welfare). But within that they create mechanisms to achieve many smaller goals. At the low level, an instrumental goal might be 'send this email' or 'win this argument with a colleague' or 'sell more of widget K' or 'teach a new colleague how to schedule a meeting'.

Similarly in my life, a final (or at least high-level) goal might be 'find a satisfying career', and an instrumental goal to that might be 'find an area where I can get care about my work enough to want to get better', and an instrumental goal to that might be 'learn organic chemistry', and instrumental goal to that might be 'read the chapter on alcohols in the textbook'.

But instrumental goals are not final goals. The first plan you came up with might not be the best; as you learn more you'll want to revise your view about which instrumental goals will lead most effectively towards your final goal. Even if you do have a good plan, you need to know when to move on from achieving one goal to achieving the next.

But humans, and corporations, have finite cognitive capacity. Some instrumental goals are big enough that to achieve them, you have to build your whole self into a machine focused on that goal. The danger is that at that point you start to think of that goal as a worthwhile end in itself; you forget that it was only a means to a broader purpose.

Most of the world's problems come from forgetting the final goal. Institutions develop instrumental goals of self-preservation, so coal companies keep existing, even when they are harming humanity's broader goals. And institutions delegate work to low-level employees who never saw or wanted the higher-level goal in the first place. This is related but not quite the same as principal-agent problems, because you often don't want your low-level employees to fully share your goals or to think independently about how to optimize them; you need them to be legible and you need them to cooperate in a broader strategy.

It's tempting to see career 'advancement' as a goal: getting promoted, leading larger groups, executing bigger projects. Many people (especially in finance) view making money as the goal. But these are at best instrumental goals, and they are useful only if you remember what your larger goals are and use them as stepping stones towards achieving them:

• If making money is an instrumental goal towards living a comfortable life, at some point you should remember to stop and actually enjoy your life. If it's an instrumental goal towards doing good in the world, you have to remember to actually do that.
• Acquiring power is worthwhile only if you know what to do with that power. As a politician, at some point you may have to sacrifice your power to actually execute the change you wanted to see in the world (eg Obamacare).
• Getting promoted is valuable only if it gives you the ability to implement some change you want to see in the world, to do some work you want to do. There's no point in becoming a manager just for the sake of becoming a manager.

The world is full of zombie machines for optimizing instrumental goals. Some of them are people, some of them are institutions. They may no longer be connected to any broader, higher goal.