intelligence forklift

Boaz Barak writes in GPT as an "Intelligence Forklift." that language models seem to function effectively as tool AI that can augment a human or artificial agent. He uses the metaphor of a forklift as a tool that augments the strength of whoever uses it:

if the vast majority of an agent’s intelligence is derived from the non-agentic “forklift” (which can be used by many other agents as well), then a multipolar scenario of many agents of competing objectives is more likely than a unipolar one of a single dominating actor.

I think there's a lot of evidence for this perspective, beyond just the emergence of language models. When we think about the capabilities of humans, there is very little that we can identify as unique to any individual human. Humans do specialize, but most people can do most sorts of things at least a little bit. We all have largely similar capabilities because we work from the same pool of "forklifts": all the products of civilization, the things that are passed down over generations, that differentiate us from a primitive human, what we might refer to as "technologies" in the very broad sense. These include

• Physical tools including hand tools, furniture, appliances, buildings, vehicles, infrastructure, the "built environment" generally.

• Formal models. One can view the development of modern science as coming up with good models of various aspects the world: the standard model of physics, the periodic table of chemical elements, genetics and evolution, human anatomy and physiology, economics and game theory, probability theory and statistics, and so on. Models can be mathematical or conceptual, but the common thread is that they provide explanatory power to help us reason about the world and design new technologies. They are the "shoulders of giants" on which we all stand, intellectually.

• Culture: including religions, myths, social conventions, management styles, folk wisdom.

• Open code: people and companies don't build modern computer systems from scratch. They rely on the huge amount of work that's already gone into developing common components: operating systems, databases, network protocols, etc. Today one engineer can put together an online business would have been beyond the capacity of all of IBM just a few decades ago.

• Open data: humans learn a great deal from our shared information commons: the Internet, Wikipedia, Youtube, media, libraries, textbooks, etc. We all have some unique experience, but much of what we know about the world is learned from the same curricula and media sources. Similarly we would expect data-hungry learning AIs to learn from as much open data as is available.

• Simulators: interactive models that may be explicitly designed (board and video games, driving and flight simulators, military war games) or learned from data (e.g., base language models), or of course various hybrids of these. These are the extension of training data to the setting of sequential decision-making.

• Market services: the capitalist economy provides a huge range of levers that anyone can pull to accomplish a task without the necessary in-house expertise. Startup companies can outsource benefits, payroll, server infrastructure. People can outsource growing food, building and repairing homes, caring for children and the elderly, and so on. Even services that are not directly in service of some cognitive task, can enable cognitive tasks by freeing up resources that would otherwise be needed elsewhere.

All of these tools allow modern humans to do things that would have been impossible for cavemen, despite having fundamentally the same kind of brain running presumably the same basic algorithms. Our accumulated library of cognitive and physical technologies, covering a range of problem spaces, is complementary to whatever "general intelligence" we possess to orchestrate these tools, and helps us in practice act much more effectively across a much wider (even more general) range of situations.

In humans, the long arc of capabilities advancement comes mostly through explicit scientific models and tool-building: things that we can communicate and pass on between generations. The process of growth is an iterated cycle of developing cultural resources (models and tools) that create a new context in which to train a new generation of minds, who develop new resources, and so on. The minds get thrown out every generation, but the technologies accumulate.

I think we can further divide these resources along a line that we might describe in various ways as

• knowledge vs tools
• intangible vs tangible
• used at train-time vs used at runtime
• data vs code
• passive vs active

with the accumulation of science, culture, and data in one category of "knowledge" (things that a mind can learn) and physical tools, market services, and reusable code in the other category of "tools" (things that a trained mind can use). This distinction is not absolute --- passive "knowledge" can include procedures to accomplish tasks, while active "tools" will still have blueprints and instruction manuals --- but

AIs will be able to use all the same technologies that humans do, and more besides. Language models were our original example of an intelligence forklift, and although humans can use language models in a rudimentary way, they can't directly graft them into their cognition the way that an AI agent can. Pretrained models for language, vision, and other skills may be valuable in a range of different AI systems, and can interface directly in terms of "thought vectors", a route not easily available to human users (barring great improvements in brain-computer interfaces). So we might expect that AI systems will be even more modular, less monolithic, than humans are.

Most of historical AI research can be thought of designing technologies --- algorithms for search and planning, probabilistic reasoning, supervised learning, game playing, causal inference, etc. --- that each address a specific class of problem. None of them are general-purpose intelligence in themselves, but all are available to be used by general-purpose intelligences in appropriate circumstances (for example, many humans now rely on planning algorithms in for their day-to-day routing via tools like Google Maps).

When we think about future AIs self-improving, should we imagine this process as continuing to add to a toolbox of modular, reusable capabilities? Or as the refinement of an inscrutable pile of insights contained within the weights of a single neural net?

What we see in humans is that both of these processes are important. John von Neumann stood on the shoulders of giants: he used language, mental models, modes of reasoning, etc. developed by previous generations, but he also was able to distill those things into an idiosyncratic set of internal representations and habits of thought that was very effective at finding deep connections and solving technical problems. And by doing this he was able to contribute whole new tools of thought (computation, game theory, etc.).

We might see this accumulation of specialized technology as a direction of progress orthogonal to the generally intelligent "minds" that use it.

will capabilities mostly advance through the accumulation of tools and technologies? or will they advance by scaling or improving the minds that use the technologies?

when we think about using AI to accelerate science, we are really looking for it to produce legible, formal models. when we think about AI mathematicians, we expect them to prove theorems. mathematics advances through the proving of theorems. an AI mind with amazing mathematical intuition, that can tell us with high statistical accuracy which statements are true or false but can't tell us why, can't prove its claims, is ultimately not going to get very far.

on the other hand, in "softer" domains like human affairs, clean formal models will be elusive. but we will have training programs, curricula, and simulators that will generally improve over time.

what do the minds add on top of the technologies?

• for knowledge, the mind is the means by which it becomes active
• the mind (and specifically a policy) is where many models must somehow be reconciled
• when we train a transformer, it compresses the data; it finds patterns and connections. this modeling or simulator aspect is itself a new technology that can be used by other agents.
• minds compress not just declarative knowledge but also procedural knowledge, how to do things. von Neumann doesn't just have a deep map of mathematical results, he has great intuition for how to do math, what sort of questions are worth asking, etc.

One direction of improving AI is to train larger and larger multimodal models that compress more and more information and skills into a single ineffable blob.

• dense blobs don't involve conditional computation and are clearly inefficient
• sparse/moe blobs that do involve conditional computation can still be blobs. they are not necessarily 'modular'. but there is a tendency towards modularity, since conditional computation necessarily means there are different 'pieces' of the model that can run or not run.

contrasts:

• dangerous technologies (nukes, bioengineering, nanobots) usable by any agent, versus dangerous agents

What I don't understand:

• what is the core of a generally intelligent mind?
• distinction between formal models and mental models. mental models can be conveyed as linguistic forms, but in their core sense they are intrinsically mental, formed from scratch in each new mind as it distills its experience of the world (including any formal models it studies).

Even in people, the notion of a unified agentic self is an illusion; the self is a construct. We are collections of parts.