theory of intelligence

tl;dr: the ideas we need to build intelligent systems may be different from those we need to understand them. Both are important, but for any given idea we should try to be honest with ourselves which category it falls in.

Suppose that in the next few years someone trains a giant neural net for a very long time on a large multimodal data set, and the resulting system passes a Turing test and can reliably surpass humans at general intellectual tasks. Will we have 'solved AI'?

In some practical sense, yes. We'd (by assumption) have a proof of concept solution to the engineering task of AI.

In this hypothetical world it would turn out that many of the concepts explored by AI researchers over the last several decades (very incomplete list of examples: learning theory, Bayesian decision theory, explicit search, meta-reasoning, understanding language and linguistics, work on dual-process cognition, on AI safety, on concept formation and abstraction, the structure of ideas, hierarchical planning, game theory and mechanism design, markets, optimization, statistics, causality, self-reference and strange loops, consciousness, etc.) may not have been on the critical path to actually building AI. Would all of this work have then been just a waste of time?

No, because building a thing and understanding it are different problems. A given area of conceptual work may not be central to the eventual technology for producing intelligent systems. But it may still help us better understand, harness, and extend such systems.

By analogy: studying psychology helps us understand human minds, despite having no direct impact on the biological mechanisms that produce them. Understanding human minds helps us organize our societies, treat mental illnesses, and brings us closer to God.

Similarly, concepts that help us understand artificial minds will be useful even if they are just maps and not the territory itself. Maps are important. Understanding the behavior of intelligent agents is important even when humans are the only intelligent agents in the world. It will only be more important when we have capable alien intelligences working among us.

Good theory can also help sharpen our engineering. People were building bridges for a long time before we had solid explanations for why they stood up, but modern engineering theory has made it possible to reliably build much more significant structures. The theory in itself might not be sufficient to actually build a good bridge: there's a lot of metis in any engineering practice, and the actual technology matters (if you could only have one, would you rather have a perfect understanding of Newtonian mechanics, or unlimited access to cheap high-quality steel?). The theory isn't always the most important thing. But the combination of good theory and good engineering makes possible new things that wouldn't have been possible with either alone.

Note that 'theory' doesn't have to be formal mathematical theory. Of course, mathematical theory has obvious value, but even more fundamental is having the right set of concepts and a framework for reasoning about them. For example, concepts like supply and demand are important in economic theory even though they are not inherently formalized (of course, one can build formal models of them, but the model is not the thing itself: we understand that there can be multiple formal models of the same thing, and it's possible to debate which of them best captures various aspects of our understanding).

The synergy between theory and engineering works in both directions. Currently the most promising work in AI is in my opinion the engineering work done at places like OpenAI, which is throwing resources at the problem of building intelligent systems. This may have massive practical impact. It may also be highly important for the theory of intelligence: it gives us artifacts to study, and its success will dramatically enlarge the 'market' for theoretical work by opening up new avenues for real-world impact.

There is still quite a lot of disagreement in the AI research community over which ideas will be concretely useful in building AI. My personal view is that computation is important, so frameworks like Bayesian reasoning that abstract over computational details are unlikely to provide the right base-level principles for building intelligent systems (though they may still have significant explanatory value). Others may disagree, but as a researcher it's important to be honest with yourself where you think the topics you're working on will fall along the practical vs theoretical spectrum, and the degree to which each type of work motivates you.