general intelligence

Is there such a thing as 'general intelligence'? What capabilities does it require? Is it a goal worth striving for?

We usually speak about human intelligence as 'general' by comparison to machine intelligences that are clearly task-specific: chess engines that only play chess, speech recognition systems that only recognize speech, etc. Humans can do all of these things, or learn to do them, and can also adapt to new situations and tasks that have never been encountered. The ultimate goal of artificial intelligence research is sometimes framed as producing an 'artificial general intelligence'

Possible characteristics of general intelligence:

1. world models: humans attempt to model the 'real world' and everything in it, not just an artificial domain like a chess board, or a limited physical slice like a Roomba that maps your living room. I'm not sure there's a hard boundary here: no human really attempts to model the whole world, and you could imagine various intermediate domains: a military strategist doesn't have to know much about art, and a mathematician might not have to know much about the physical world at all. But good human strategists and mathematicians can think effectively and creatively within those domains, and could at least in principle be trained or retrained to work in other domains.
2. In particular, humans are embedded agents: the world we model is the world that we ourselves exist in. So we can reason about how our actions affect our own selves and our future capabilities.
3. Reflexivity: humans are self-aware, we have models of ourselves and of our own thought processes, and are capable of meta-reasoning; we can introspect and try to improve our thinking.
4. Task-agnostic: humans can (learn to) perform tasks that we were not originally designed or trained to do.
5. Internal motivation: humans can to some extent 'choose our own goals', rather than blindly following an external loss or reward signal.
   • I'm not sure I like this framing, because it conflates choosing instrumental goals, which even a non-general system could do, with choosing 'final' goals, which is totally ill-founded following the is vs ought dichotomy. But we clearly have the ability to pursue very abstract, flexibly defined goals and to appropriately refine or subdivide them as needed.
   • Debatably the notion of final goals is bound up with moral realism and conscious valence. If valence is the ultimate final goal, then humans with conscious experience have direct access to it in a way that AI systems may not.
6. Continual learning: humans are not cleanly divided into 'training' and 'inference' phases. As we act in the world we continue to learn from experience and adapt to new situations and events.
7. Self-modifying: our learning from experience is not purely mechanical. We can and do reason about what lessons we should learn and what strategies / goals we should pursue.
8. Scalability: tasks too big for a single human can often be solved, eventually, by the emergent cooperation of tens, hundreds, thousands, even millions of humans (albeit with substantial overhead and inefficiency in the case of most large organizations). This ability to cooperate is fundamental to most human successes. Language (or shared abstract representation in some form at least) seems essential to this.
9. Universal execution: given sufficient time and resources, a motivated group of humans can accomplish any goal that is feasible-in-principle and well-specified.
   • 'Sufficient time and resources' is doing a lot of work here. An entity with truly unbounded time and energy could accomplish a lot of things by brute force (e.g., solving NP-complete problems) but we might not consider this to be an intelligent approach.
10. Universal representation: human minds can 'understand any idea', even, e.g., in novel areas of math where we have little natural intuition.
    • Of course, if there were ideas that we fundamentally can't conceive for some reason, it's not clear that we would be aware of this.
    • Many humans do have trouble understanding some concepts, especially of more abstract or mathematical flavors. Some concepts are inherently complex and may require some minimum level of intelligence to grasp.
11. Universal computation: minds are Turing complete; we can simulate the execution of arbitrary computer programs (at least up to resource constraints). This seems like a clearly necessary condition for general intelligence, in that whatever policy architecture we use must have the ability to represent (and potentially learn) arbitrary procedures, including meta-procedures like a universal Turing machine. But it's clearly not sufficient on its own.
12. writing: related to the previous point, part of simulating a Turing machine is the ability to store state: the contents of the Turing tape, which models the scratch paper used by human mathematicians. A human who can't write can't simulate most Turing machine, and so is not a general intelligence in some meaningful ways (and neither are nonhuman animals, by the same token). Similarly, any generally intelligent AI will need some equivalent to writing, e.g. access to a large, controllable memory (c.f. transformers with memory).
13. Human interaction: a general intelligence should be able to operate in human society (humans have an unfair advantage here, since at least some of our social conditioning is probably 'hardwired' by evolution). It would need to learn how to maintain relationships: what people want, how we bond, how we entertain, comfort, inspire, lead, and love each other. An 'emotion-general' agent that could 'only' do all of this, but not, e.g., difficult reasoning tasks, might still be incredibly capable (especially in a world where other types of agents exist to help with their shortcomings).
14. consciousness: in general intelligence is not consciousness; you can have either without the other. There are hard jhana states of intense pure consciousness that contain no thought whatsoever, and conversely one can imagine extremely capable reasoning systems (e.g., Deep Blue) having no conscious experience whatsoever. But even sidestepping the hard problem of phenomenal consciousness (whether there is "something it is to be like" to be a given system), consciousness seems to correspond to a certain information architecture that combines flexible attentional processing with a general representational capability (our "conscious state" is a unified representation of perceptual information from many modes) and a reflexive capacity (we are aware that we are aware).

Could there be an intelligence 'more general' than humans?

Certainly it's not hard to be more general than any specific human - we all specialize quite a bit, and even if we are quite adaptable, it takes time to learn new capabilities, and no one human can ever learn all the capabilities. This is true even in restricted domains like math: no modern mathematician is truly expert in every area. But there is variation, and we occasionally have polymaths like von Neumann with unusually wide-ranging expertise; one might say that such people are in a sense 'more general' than the average mathematician. In the same sense, a sufficiently large AI system cross-trained on many tasks could be a super-polymath, able to seamlessly integrate capabilities that would be beyond the reach of any single person.

Since individual humans can be quite limited, a lot of the case for human generality relies on scaling our capabilities via coordinated action of large groups towards common ends. Increasing the scale of our cooperation, from primitive tribes to modern corporations and nation-states --- what we might call civilization --- has been one of the chief drivers of human progress. But scaling human organizations is difficult and inefficient; our communication bandwidth is limited and organizations are beset with principal-agent problems. Systems that could genuinely 'mine-meld' and subvert their own will to a common goalConnections to AI safety and corrigibility? might scale better than humans and thus be more effectively general in practice.

What would the 'least capable' generally intelligent system look like?

Some cases to consider, for intuition:

1. Nonhuman animals. I think we would mostly consider these not 'generally intelligent', insofar as they don't have the capability to learn language, program computers, etc. But animal intelligence can certainly be 'flexible' in a range of ways that current AI systems struggle with.
2. A single human being of average intelligence.
3. A single human being of top-level intelligence (e.g., von Neumann).
4. An untrained large transformer + SGD.

An operational definition of general intelligence

We might say that a system has fully-human generality if it can learn to do any task currently done by humans, given training analogous to what humans receive.

For example, such a system could reliably learn to drive a car at human level, given ~20 hours of explicit instruction and feedback, and a few hundred hours of practice (likely on top of a substantial pretraining process to instill a reasonable world model). Or it could learn to practice medicine at the level of a human doctor, given coursework and several years of supervised practice equivalent to the standard training program for human doctors (med school followed by residency).

This is obviously a human-centric definition, and embodied tasks would require a sufficiently capable robot, so it is not a satisfying definition of general intelligence in the abstract. But it is at least a sufficient condition for economic transformation.

Some skills, like political leadership or scientific research, are not reliably trainable even for humans. For these rarer skills we'd require the AI to at least match human learning capacity. It might be that not every combination of innate qualities (for the AI, perhaps a pretraining run and/or random seed over "personality" initialization) and training program succeeds at developing the skill. But if we spin up 100 copies of the agent and enroll them in 100 PhD programs with different advisors, we should get at least as many successful outcomes as we do with human students.

One tough thing about this definition is that it might take a long time to know whether we've satisfied it. A system with human-level learning capacity still needs to actually do the learning, which might take a long time, as Sam Altman pointed out in 2016:

OpenAI was focussed on having its system teach itself how things work. “Like a baby does?” I asked Altman. “The thing people forget about human babies is that they take years to learn anything interesting,” he said. “If A.I. researchers were developing an algorithm and stumbled across the one for a human baby, they’d get bored watching it, decide it wasn’t working, and shut it down.”

It would be more satisfying if we could somehow identify some abstract set of learning capabilities that are Platonically "complete" for a general agent, so that we'd know that a system implementing them would be able to learn human-level capabilities, without waiting for it to actually happen. But given how far practice has gotten ahead of theory in ML these days, this seems exceedingly unlikely.

Are language models generally intelligent?

Compared to previous AI success stories like AlphaGo, large language models seem like a qualitative leap in generality and flexibility.

Modeling language is not exactly modeling the "real world". But LLMs model our descriptions of the real worldLanguage is interesting because it contains both descriptions of the world and actions in it, so a language model is not just a world model but also a policy. Collapsing these functions into a single network feels likely inevitable for any general-purpose agent (projection is unavoidable)., and do so in an extremely domain-general way.

In some sense, a large model is as general as its training data. A system trained on only code, or only English text, is less general than one trained on both. A system also trained on images or video would be more general, in a different way: it might learn capabilities to incorporate visual concepts, to reason visually, and of course, to describe and even generate images. We can generally give a model a stronger and more generalizable understanding of the world by training it on more data, across more modalities.

However, all of this is somewhat orthogonal to what we mean when we call humans "general". Most of us would say that people from the 1800s who never saw computer code were still generally intelligent, and that blind and deaf people are generally intelligent despite missing an important sensory modality. Our intuition is not so much about what people have learned, but about the capacity to learn and to reason about what has been learned.

In LLMs, the capacity to learn lies in the transformer + SGD architecture. And we have strong evidence that this architecture is capable of learning linguistic skills across many domains, as well as (with a few additional architectural ideas) excellent models of images and video. So we could say that large autoregressive transformers seem to be domain-general learners. But they certainly don't (yet) have the capacity for general agency, reliable reasoning, and self-reflection that we'd expect of a human-level general agent.

Is 'general' intelligence even worth caring about, as opposed to other potential frames like 'flexible', 'helpful', 'transformational', 'controllable', …?

There is a real sense in which LLMs are more general than anything we've had before, and this feels like a compelling way to describe what makes them exciting.

Fully-human generality seems like a useful lighthouse goal. But the reference point is not some abstract notion of generality; it's actual human capabilities. We look at what human tasks we haven't yet succeeded at teaching machines to do, and try to figure out how to modify our most general machines to encompass those tasks as well.