A 'natural' abstraction is one that we expect any agent (or at least, a wide range of agents) to develop because it gets at something…
When thinking about the [ reward ] function for a real-world AI system, there is always some causal process that determines reward. For…