Created: June 19, 2020
Modified: April 23, 2021
Modified: April 23, 2021
AI research landscape
This page is from my personal notes, and has not been specifically reviewed for public consumption. It might be incomplete, wrong, outdated, or stupid. Caveat lector.- As of April 2021:
- Giant transformers work better than anyone has a right to expect. GPT3 is fucking amazing. DALL-E clearly has some level of 'understanding' and is generalizing. CLIP is a preview of a world where we really can program computers with natural language.
- AlphaFold is a real scientific revolution.
- MuZero and its predecessors / successors are insane. They become superhuman at (perfectly simulatable games with perfect information) pretty much automatically.
- AI research is not slowing down. Hardware and training infrastructure are getting cheaper.
- Will we plateau? Will we max out at 10x or 100x GPT3, and find that even all the data in the world isn't enough to get human-level AI from giant transformers?
- We might not. What would this imply?
- Novel applications of AI will be large industrial projects requiring significant investment. But AI is so fucking transformative that this investment will be forthcoming.
- This would be the practice of AI outpacing the theory of intelligence.
- If we do, what will we be missing?
- Planning and intention. There's a sense in which we plan our language use, and to speak well, GPT3 must learn to do something analogous. But fundamentally, the language modeling task is behavioral cloning. There's no notion of intention, goals, decision-making, information gathering, deliberate practice to improve skills. It might turn out that the most efficient way to compress large datasets involves some of these concepts. But to build AIs whose behavior we can steer, who can solve problems that humans never thought about, who are guaranteed to be friendly, we might need more explicit mechanisms for agent-based thinking.
- We might not. What would this imply?