Created: April 07, 2020
Modified: April 07, 2020
Modified: April 07, 2020
neural nets do work
This page is from my personal notes, and has not been specifically reviewed for public consumption. It might be incomplete, wrong, outdated, or stupid. Caveat lector.- Like the proverbial half-full glass, smart people can look at the same reality of the current capacities of neural nets, and come to opposing takes even as they don't disagree on any hard facts.
- I wanted to write this about how I disagree with many people. But I realize I'm struggling to sustain the take. I'm creating a distinction that doesn't need to exist---I'm thinking of there being people 'out there' who don't care about AI for AI's sake, who never really 'got it'; they're just in the field because they were smart and it was a prestigious and analytical thing they could do. On the other hand, I did care deeply about AI and that makes me special.
- I don't need to divide myself from other people in order to feel special. It doesn't matter what I think I know about other people's motivations. I'm probably wrong in most cases---I can't see their internal lives any more than they can see mine---and in any case I should take them at their word. They're interested in what they're interested in, even if it doesn't "feel the same" to them as to me.
- The thing to remember is that there's a side of me that gets excited about AI. Most of the time recently I've been grumpy about it---I look at the AI community as mostly BS, doing research that will never be useful, arguing over stupid crap---and I think that's partly a reaction to no longer being in that community, and being surrounded by more practical people at Google.
- But it is really cool that a computer can learn to recognize images by 'experiencing' batches of them over some training period. It's cool that RL algorithms can learn to walk like a child or an animal. It's cool that Transformers can ingest a ton of text and spit out coherent, structured lengthy paragraphs that can answer questions, do translation, and more.
- There's the old saying that once we know how to do something, it's not AI. I feel that way about modern ML and deep learning now, personally. It used to be exciting to me because it seemed like magic. Now that I know more of how it works, it all seems limited and unremarkable.
- I worry that that part of me that thinks that AI is deeply cool was motivated, at some level, just by desperately wanting to know 'the solution' for how to navigate the world. And that part was misguided, first, because the problem is hard; but second, because knowing the solution is a very small part of living the solution.
- I also worry that my interest in AI and computers in general grew out of a lack of interest in or connection to people. It was kind of the 'original sin' and kept me from getting the experiences I needed to develop a confident personality.
- But I remember that I had other motivations. I thought computers were cool, because they just are. It's a machine that can think! It's easy to notice that thought is incredibly powerful, because humans dominate the modern world. It's a natural connection to realize that computers will be powerful. It's natural to see the upwards trend, with fresh eyes as only a young person can, and grab on to it. It was natural for me to be interested in math and science, and for computers to be a natural extension.
- And moreover---if the causality I'm worried about is that caring about computers caused me to care less about people---it doesn't matter whether or not it was true. It doesn't have to be true. Most people I know from grad school do care deeply about their work but also about human relationships. (and to be clear, I care quite a lot about relationships; I just don't express that in my actions). This is an area where I need to adopt a growth mindset.
- Back to neural nets. I often find myself being the expert who has to shut down people's fantasies around neural nets. I find myself being cynical about the future of AI research. But at a very deep level that's not who I am. I don't need to change my knowledge of any facts, but I need to remember to be excited. It feels 'mature' for me to be 'realistic' about what can be done. I've realized that I personally won't be able to do grand things on the scale I'd wanted to---and that makes it harder to imagine that those things are possible. I can be realistic while still being excited. They key to realize is that success means having plans that you can make immediate progress on, while also continuing to be attracted to the lighthouse off in the distance.
- 'Neural nets work' was the wrong title. It had the right connotation for me inside my mind, but written out, I think 'Neural nets are exciting' or 'We should take neural nets seriously' might be better. We shouldn't see neural nets as just a collection of logistic regressions, or a parameterized function class. We should remember that they're the closest thing we have to artificial brains. Even though the mechanisms are much simpler than real brains, they can still do a lot that was inconceivable fifteen years ago.
- It's wrong to say that what they're doing is not 'learning' because it's not the same as the brain---to do that creates an unbridgeable gulf between men and machines; it requires belief in some sort of duality, which is more of a religious than scientific position. But we need to develop language for how to measure learning, what milestones are important, and we need to understand where we stand regarding these milestones. Many of the 'technical' details of tasks in NLP or vision---fields that I've written off sometimes as 'too specific'---actually express a lot of insight in being framed to test hard-but-tractable areas of the frontier of learning techniques.
- I think this note as a whole gets at something I feel more generally: it's easier to be motivated when you have a mental picture of the path to the goal. If I think of intelligence as a thing I 'understand' at an intuitive level, then I can see myself continuing to make progress in thinking about it, and then I'll have the motivation to do the long hard work. If I think about it as a 'bottom-up' research endeavor, where we are pursuing random directions because we don't really know the way, then it's not motivating. But I have sometimes felt like I know the way, and there's still a lot of me, buried somewhere but maybe not too deeply, that does.