Created: January 27, 2022
Modified: February 10, 2022

blog posts to write

This page is from my personal notes, and has not been specifically reviewed for public consumption. It might be incomplete, wrong, outdated, or stupid. Caveat lector.

writing inbox
the privilege of having correct advice work out for you (the privilege of advice working out)
- Let's stipulate that any broadly-scoped advice won't work for everyone. "Get a therapist" might be good advice in general, but some people won't be helped by it. "Avoid socializing and focus on your studies" might be good advice for most high schoolers, but what some people really need is more social immersion. "Do what makes you come alive" might be generally good advice, but for some people this is disastrous as they discover there's no support for what they want to do, no one to help them grow, etc. "Don't worry about classes, focus on research" might be good advice for many first-year CS grad students, but there are some students who really do need the intellectual foundation from coursework and for whom venturing into research without it will just leave them lost and wasting a ton of time.
- People who have received good, sensible advice and followed it to great success are well set up to be mentors and leaders. They can repeat that advice to mentees, with the confidence of success. "Here's what worked for me." The mental framework they inherited for reasoning about the world worked out, and so they feel justified in passing it on to others. The advice was sensible both a priori (as most good advice is) and has born out in practice.
- But people for whom the advice didn't work, don't have the privilege of passing it on. Suppose that "Don't worry about classes, focus on research" really is the correct advice for a sight-unseen grad student. If that advice worked for you, you can authentically give a grad student that advice. If that advice didn't work for me, I can't pass it on. I can only pass on the counterfactual: I think it would have been better if I'd taken more classes. But I don't know if this is even true. Maybe that would have been even worse, and I would have never found understanding and also never learned any research skills.
- (could also write this with NameRedacted's "you have an ordering problem, not a selection problem" advice about a job search. it might be good general advice, but bad for me.)
sharing inner lives
- not sure if I'll ever be down for this. but part of me wants to post the diarizing and thoughts stuff online, somewhat verbatim. I'd replace most names with generic XX, and particularly important (recurring) names with J or S or whatever. I'd do it partly as an exercise in rereading and reflecting on the entries, and would make notes as to how they read to me now.
- pros: it's an exercise in radical intimacy. the shells we expose to the outside world conceal all the turbulence, insecurity, and rich complexity of our inner lives. I know in principle that other minds exist, like mine in many ways, but these days that belief is so rarely validated by evidence. Facebook profiles, academic CVs and twitter, professional interactions, are all highly constructed and performative. Even in-person friendly interactions I'm not going to dump my brain-state on you, that's way too much to handle unasked-for from a stranger.
- cons: this is basically a teenage livejournal. no one cares about my drama!
  - and, people who might have respected me, wanted to learn from me, follow my example, might be turned off by the stupidity and pettiness. and bad writing style.
  - and, the right way to bare my inner life to people is to build friends and relationships, not just give it away to the internet.
on confidence, research, masculinity, sexual identity
- (would have to flesh this out a ton. but basically, a lot of research is useless, and to sink time and effort into it anyway requires a certain level of unearned confidence. this comes more easily to men, and especially straight men who have spent less time questioning themselves or being wrong about their own future selves or desires)
- and this relates to general workplace conversations. who becomes a confident executive? who leads a team direction? not necessarily the person who is most right, but the person who speaks the loudest and most confidently.
- changing this dynamic is one important reason for inclusive workspaces. and inclusive academic spaces.
- (of course the flipside is, we should learn to take pride in our ideas and express them confidently. advisors should help mentor us in doing that.)
- (as usual this is probably many posts)
notes on a job search
- it's easy if comparing essentially similar positions. it's hard if comparing existentially different views of future life.
- parable of the candy bar
- every decision feels like killing a part of my self
- advice: nothing is permanent
  - scott: "swim towards the warmth"
  - NameRedacted: you have a choice problem, not a selection problem
technical notes on a job search
- what are the 'next big things'? what did I learn from Vicarious, Freenome, Uber? (and other places I looked at but didn't apply to… let's not be too selective).
why is bayesianism the next big thing in AI?
- answer: generalization
- what are the problems?
- learning from sparse rewards
- learning from unlabeled data (basically the same as previous)
- counterpoint: sure Bayesian inference may be a way of thinking about the right computation to do. But we could also just try to learn the right computation directly, through gradient ascent on differentiable circuits (deep networks). And if some form of (approximate) Bayesian inference in some implicit or explicit model is the right thing to do, we'll discover that directly rather than needing to hand-design something crazy. response to this: sure it's plausible in some cases (see monference). but if bayesian inference is the right thing to do, you don't want to have to relearn that fact from scratch on every problem. we can think about building neural architectures inspiried by the structure of bayesian computations. but we also often want explicit models. because they let us interpret our inferences, gain confidence they are physically/scientifically/causally meaningful, and allow systems to be easily improved by domain experts.
what is bayesianism?
- conflation of models and uncertainty. bayesianism is not technically the same as generative modeling -- you can be bayesian about the params of a discriminative model -- but philosophically they are closely tied. Bayes' rule inverts conditional probabilities, so it converts generative models into discriminative prediction rules.
  - there are arguments for generative models: for writing down the structure in the world leading to your data, making your assumptions explicit, expressing them in a way that is interpretable, captures the real dynamics of the data-generating process, allows for strong generalization, combining information and answering many different queries from a single principled model, or performing many different tasks (changing reward fns).
    - (this is basically the argument for probabilistic programming)
  - separately there are arguments for predictive uncertainty. knowing the confidence in your predictions is decision-thereoretically important. how sure are you the object in the road is a person? is it worth slamming on the brakes of the self-driving car? it's also important for information gathering, exploration vs exploitation. when am I sure a translation is correct, vs when should I explicitly ask a human for clarification? Even internally for allocating computation: when am I sure I have a good response to this go/chess/Starcraft move, vs when should I spend more time thinking?
  - these arguments are sometimes lumped together inside of general bayesianism. but they are separate! you can have generative models fit via non-bayesian methods. and you can get predictive uncertainty from discriminative models, and even discriminative models where you are not explicitly bayesian about parameters. (philosphy of aleatory and epistemic uncertainty aside, in principle a proper scoring rule should train a non-Bayesian network to output calibrated probabilities, if it has the capacity to do so).
  - HOWEVER they are a very natural fit. Generative modeling without Bayesian inference usually fails because if you have any interesting set of latent variables, you need to integrate over them. Your problem is never really MAP, but marginal MAP, which means you're doing all the work of being Bayesian anyway so you might as well go all the way and get the benefit.
  - what about uncertainty from discriminative models (and even non-Bayesian discriminative models)?
    - questions about extrapolation? adversarial examples show that non-Bayesian nets do arbitrarily bad things once you leave the training manifold, including being hyperconfident about wrong answers.
is bayesianism bullshit?
- marr's levels of analysis: computational, algorithmic, implementational.
  - we can think about how the system is implemented independently from what the system is 'trying' to do.
  - but the brain isn't 'trying' to do anything. it was not designed; it's just an artifact.
  - at some level the goal of AI is the implementational level. evolution didn't derive the brain from first bayesian principles. it just built a messy implementation that worked. we can come up with abstractions that capture a lot about what the brain does. but abstractions by their nature erase certain aspects of the system. bayesianism erases computation.
- bayesians see something like a NN that doesn't maintain parameter uncertainty as 'wrong', and want to fix it but adding uncertainty. Which is an interesting exercise, but as a dogmatic response it often misses the point. because adding uncertainty makes the computation more difficult (we cannot do exact bayesian computations on neural nets; there are approximations but the good ones like HMC are infeasible and the feasible ones are approximate enough to be hard to reason about).
- situations where computation and bayesianism are compatible! noise as bayesian inference. rather than representing uncertainty with careful explicit calcualtions, let the messiness and unreliability of implementations implement uncertainty on our behalf! examples include dropout, SGD as inference, low-precision float representations as regularization, etc.
how to train academics (/advise grad students)
- I think I have a perspective here that is useful. I'm very close to the process, have thought a lot about it, seen what works in other students. I'm sure senior professors who have been doing this for many years have convinced themselves they are good advisors -- the primary skill of academia is convincing yourself you are good at things. But in practice, many advisors are bad. Some people deliberately make this tradeoff, they choose to define their job as advancing their own research agenda, or maximizing paper count, or contributing to public understanding of their field, or teaching undergrads, or many of the other facets of professorship -- and the cost of not focusing on advising is that they are bad at it, and this is (from their perspective) a fair tradeoff, because you can't be good at everything. However taking grad students is a commitment on both sides, and it's fair to decide you don't want to focus on advising students, but if you do take students you have an ethical commitment to try to be a good advisor for them. they might not all become top researchers but at least you should try to avoid ruining their lives. And that's hard, and hopefully these thoughts could be useful.
- most students come in thinking very myopically (they want to 'do
  - well', 'publish papers', gain status and reputation). part of the
  - job is to develop a real research program, the ability to think
  - more broadly about how to push it forward.
- for example: writing a paper with the idea of getting it past
  - reviewers. that's not really the point! you want to write a paper
  - that people will read and be influenced by. 'proof by intimidation'
  - is not enough.
- some basic themes:
  - send students to conferences early, and make sure they meet people! you want them to early on start to understand the research community that they are writing for. and get them to understand that papers are written for people to read: there is no Platonic ideal of the perfect paper, only a paper that is interesting to the people who are going to read it. starting to fit into a research community -- both senior academics, and other students their age from other places -- is also good motivation, since you see other people who will value your work, and even more crucially for a sense of belonging. Developing connections to an academic community is especially important for minority students who have spent their lives worrying they don't 'belong' --- women, racial minorities, LGBT students, etc. --- but not exclusively; straight white guys have feelings too and a sense of welcoming and community is important for everyone.
  - make sure they have opportunities to talk about research (at all stages) on a regular basis. this means cultivating a social lab environment where students are working on similar enough stuff, reading the same papers from the same community, etc., to be able to talk to each other about their work.
  - give them the chance to publish papers early. this can mean, give them an easy project to start out, or have them join a project led by an older student. It's important to publish in order to get a full sense of the process, to gain the confidence of having produced results, and to start to get a sense of the kind of research you want to do. It's easier to think about the early stages of a research process (problem selection, etc) if you've already experienced the later stages (designing and running experiments, writing, presentation, reviews, etc) and can bring those experiences to bear. and in particular, working in collaboration with an older student helps them learn all the details they won't get from you, the advisor.
  - ultimately, make sure their thesis project is their own. Josh tenenbaum says, "don't work on research you don't love". Of course everyone struggles but if a student don't love at least the idea of their thesis, it's going to be hard to push through those struggles.
  - even deeper than this: show that you care about helping them achieve their goals. NameRedacted says: "what do you want?" Every meeting will start with that until you figure it out. That can mean getting a faculty job (involving networking, etc), attacking a really ambitious problem, building useful tools, getting teaching experience, etc.
  - encourage community, both within your group and the wider grad student population at your school. hold group retreats, ski trips, bar nights, dinners at your home, whatever seems appropriate. Spend time having nontechnical conversations with your students.
should you do a phd?
- tenenbaum: 'dont work on research you dont love'
- advisor fit is crucial. it's like a marriage but more so.
- top schools are not set up to help you find your passion. they expect you to come in pre motivated. they say you can come in, try different things, search for an advisor, etc., and this is true in theory. but it's not a good gamble.
  - basically if you're coming directly from a liberal arts undergrad, you should not do a phd. seven years ago I would have argued the hell out of the opposite position, the benefits of broad thinking and a liberal arts education. but even if you think you've gotten research 'experience' at a liberal arts school, you probably haven't. unless you've been part of a multi-person collaboration (academia is a social process! something I didn't realize and another blog post in itself), published a top-tier paper, been to a conference, met people and made friends and felt part of a research community, with a corresponding idea of what you want to work on and who you want to work with. The top students at big research schools do have these experiences, and they come out more-or-less prepared to start PhD programs at top schools. Fair or not, most LAC 'undergrad research experiences' do not prepare people in this way. Research experience is not a skill, it's about joining a community. If you come to grad school without already having joined a community, and even worse if you're unsure which specific community you want to join, then chances are you will never join a community and your grad school experience will be a depressing waste of time.
- depression rates are high. the cost of a phd is not, 'work on cool stuff for a few years, then get a real job'. it's possibly, 'feel socially pressured to lean in to an abusive relationship for many years, destroy your entire identity as a smart person, fail for reasons that mostly aren't your fault but you are almost certainly the sort of person who will think they're your fault, become disillusioned about your field, destroy personal relationships, emerge extremely depressed if not suicidal, end up at age 30 with less self-confidence than you had at 22'. this is a good gamble only if,
  - a) you are really sure you've got a house advantage: a concrete research plan you want to pursue, that you've already worked on, with an advisor you personally click with.
  - b) or, if there is nothing else you can imagine doing. That is, if you literally cannot imagine any other life path making you happy. This is not a particularly good reason to imagine that a PhD will make you happy. Your future has negative expected utility, and that's unfortunate, but if this is really the situation you find yourself in you might as well try to make the best of it.
- people will counter: 'I found my passion in grad school!'. 'I had a great experience!'. 'They were the best years of my life!'. And those people are speaking their own truth. But those are anecdotes, and there is a massive selection bias from people who either
  - had the process go well enough that they are still in academia and able to engage in the conversation.
  - emerged with enough self-confidence to engage in the conversation
- And those people are right that sometimes the coin flips your way, sometimes the gamble works out. But we're all biased by our own experiences, and they underestimate how often things don't work out. They also don't understand the cost of a negative outcome, because they've never personally experienced it.
why grad school success is unpredictable
- fundamentally, you will succeed as a grad student if your research works out. if your initial projects turn out well, you'll get to write them up, you'll get the experience of doing that and going through the publication process and going to conferences and meeting people and selling your work to a community and thinking about the next steps of a long-term research program and you will generally build the confidence and skills required to be an academic.
- meanwhile if you do research that doesn't work out, you will not get these skills. if your first project(s) fail, you'll become less confident, less motivated, more alienated from the community. and again there are feedback loops here. Of course it's possible to break out of the feedback loop! People can take a while to find their passion, can persevere through failure and eventually find the amazing idea that animates and makes them successful. But they will be at a disadvantage relative to those that have the support system (institutional, reputational, and internal confidence) that comes from early success.
- but research is uncertain by definition: if you know a project is going to 'work out', it wouldn't be research! of all the directions researchers are working in at any given moment, some will pan out, some won't. of course it's possible to have opinions about which these will be, but no one is infallible.
- even many established academics spend years working on research that turns out to go nowhere. maybe you even publish a lot of papers on a project, but it turns out to not really have impact, to be eclipsed by other lines of work, to be an interesting exercise but not really part of the story of the field that gets remembered. and as a senior academic, you're okay with this. you've had enough successes to have built up self-confidence, to understand that not all projects are wild successes and taking risks is important, and you are still a smart person and still have a place in the community even though you worked on some stuff that didn't work out.
- but grad students mostly don't have this confidence. in fact it would be pathological if they did: they have no evidence they're good at research, and when a research project goes badly, the rational response to evidence is to update in the direction of lower self-esteem. It's true that some people have the pathological ability to maintain confidence even in the face of having failed at everything they've attempted, and they are disproportionately straight white males (exhibit A: our current president). Taken to extremes this is not a desirable trait, yet it is what academia selects for. To say, well if you were meant to succeed you would keep perservering even in the face of failure, even in the face of alienation from the community, having no promising research program, and no self-confidence, is a copout that denies humanity and valorizes the most pathological traits.
- so no matter how smart you think you are, how motivated, how strong of a candidate you are, as an incoming grad student you are taking a risk that the research you start working on will not work out. and as a consequence of it not working out, you will not gain the skills you'd hoped for, the validation you'd desired, the entry into the community of intellectuals you'd craved. and no one -- not you, not your advisor -- can predict this in advance. and a possible consequence of this risk is serious depression.
academics are experts on the conversation, not the thing
- you'd think that a PhD in AI would make you an expert on AI. a PhD in econ would make you an expert in econ, or a PhD in psychology would make you an expert in psychology.
- but the thing about these things is, no one is an expert in AI. we're all beginners. humanity does not know how to build humanlike AI. similarly there is a ton we don't understand about how economies work, how minds work, and so on.
- when I did my PhD I thought my goal should be to understand the technical content, to build the right intuitions about how an ultimate AI should work and what path we should take towards it.
- a lot of people were doing work that I saw as not on this path, which was therefore uninteresting. For example all of bayesian nonparametrics, SVMs, pretty much all methods in statistical NLP and pre-deep-learning computer vision.
- but as an academic you're unlikely to solve the thing (AI, economics, whatever) in your own research career. Instead your goal is to contribute to the conversation. Your goal is to know what ideas are out there, how they relate, and to do work that pulls the conversation in (what you consider) the right direction.
- this means you need some sense of direction, but that's not enough. you need to be able to discuss other ideas fluently enough to convince their proponents that you understand and appreciate them.
why AI safety is a real concern (and the deniers are also right)
- first: should you as an ordinary citizen be worried about existential threat X? probably not. your worry won't do any good. but society should work to curb risks.
- what do people worry about with AI safety? fundamental problem is value alignment*. We build systems that make decisions optimizing some mathematical objective. Because (to understate the matter) moral philosophy is not a solved field, we do not know how to write down human values in such a precise form. So whatever objective we specify will necessarily not align with the world humans actually want. As machines get more effective at optimizing their objectives, they will tend push the world into states different from those humans would have chosen. And under most existing notions of how to formulate these systems, they will be incentivized to ensure their own survival even in the face of humans trying to turn them off (footnote: people sometimes claim that the 'survival drive' is unique to biologically evolved agents. but this is false: any agent optimizing an objective function will have instrumental incentives to stay alive and acquire resources).
- The machine has some model of the state of the world, and a function that assigns values to these states of the world. The goal is to maximize the expected future value. This covers everything from robot vacuum cleaners (value = amount of dirt removed) to the facebook news feed (value = heavily proprietary, but some combination of clicks and engagement) to killer robots (value = ??? number of enemies killed?).
- The fundamental problem is that these objective functions are all poor proxies for what human beings actually want. Sometimes they are trivially gameable: a vacuum cleaner maximizes dirt removed by repeatedly dumping and vacuuming up the same pile of dirt forever. Sometimes they are more subtle: the system works most of the time, but responds to edge cases in weird ways.
- People sometimes argue that real machine learning researchers mostly don't worry about AI safety, as if that proves these concerns are unfounded. This is first of all untrue; well-known academics including Stuart Russell, …, Percy Liang, have spoken out about this problem and are incorporating these concerns into their technical research programs.
- people working on day-to-day 'AI' search are deep in the weeds of how to optimize these objectives. they are building statistical models of the world, developing algorithms to estimate the parameters of these models, analyzing the curvature of optimization surfaces, allocating computation to visions of possible futures (MCTS and alphago), … they are not incentivized to think about this problem. They see how fragile and stupid current systems are, and imagine they will always be this way: they don't believe in their own field. researchers don't always know best
- asking 'AI researchers' about AI safety is like asking financiers about economic crises. The financiers imagine themselves 'masters of the universe' but, if you have spent any time on Wall Street, you understand they're a bunch of cowboys with an instinct for their local corner of the world but have no special expertise or training in thinking about the broader effects of their actions.
- Historical solutions to the value alignment problem. In fact this problem is not new; humans have already constructed decision-making entities with power exceeding that of any individual person: governments, and corporations. In both cases we've opted against optimizing a specific objective function.
  - democracy. forces governments to respond to the will of the people.
  - markets. corporations must respond (in a loose sense) to the will of their customers, and shareholders. contrast to communism where a single objective (make more cars!) did not incentivize probing what consumers actually want.
is technology good for the world?
- people are often tech 'optimists'. Assume new tech developments are good for humanity; even if not immediately clear how. But second-order effects are hard to account for. For example, self-driving cars save lives and attention relative to human drivers. But cheap and plentiful self-driving might encourage people to stay in suburbs or forgo building good urban public transit, and ultimately lead to a worse world. The schematic is, we are currently on a local maximum, and I could do something to change the surface to 'push up' that local maximum, but by doing so I make it less likely that our dynamics (imagine something like MCMC) move to the global maximum in the near future.
- generally worth rereading/referencing Sapiens. and the argument that hunter-gatherers in general were happier than modern humans. I find this totally plausible.
- another example: facebook. It 'helps people stay connected'. Which is great. But our society has moved from one where we have a wealth of local connections to one where I now live far away from all my friends and family. And our interactions are mediated by curated social feeds rather than actually getting to know one another. This trend started before Facebook and would probably continue without it, but on the margin Facebook allows it to continue.
- a hard example: ads. certainly I have never gotten any utility from internet advertisements. They do pay for things I enjoy like Google search. But that's basically a bug in capitalism: you can get money by building something that no one wants! Yes businesses 'want' the ability to advertise but no individual person wants to see the ads.
- generally, any wealth-making enterprise that 'grows the pie' but in which most of the returns go to current capital holders. in theory this could be redistributed, but will it be?
- finally: the whole enterprise of AI. without alignment, every inch of capabilities research brings us closer to dystopia.
- similarly, biotech research. might be useful, might be dystopian. if you're optimistic about humanity, great, but that's not my current state.
- I'm not sure that human life is by default even positive-sum (negative utility). I've spent lots of time being miserable. And depressed people tend to be silent, many people live lives of 'quiet desperation'. It takes courage to kill yourself (hence antidepressants->suicide) and the evolved drive for self-preservation is pretty strong even when life is unenjoyable. So the fact that many people don't kill themselves is not an argument that they find life worth living. For this reason I don't care much about existential risk as such, or even life extension.
- I definitely don't think immortality is positive-sum. We all die constantly, even while living, and to reject that fact is the height of 'craving' (in the buddhist/sapiens framing) which can only lead to sorrow. and again, more life is only good if life is positive-sum. and even so it's not clear that the same lives over a long time period are better than many different lives coming and going.
- so what is worth doing? the first answer is, whatever gets you excited, that you can do and collaborate with people on. since the world needs people that have 'come alive'. But if you're reading this and trying to figure out what to get excited about. Some options that seem worthwhile are:
  - things that directly reduce suffering. for example, even though we all die, cancer deaths are particularly miserable. so curing cancer (and biotech research with that goal in mind) seems useful.
  - robots that directly avoid menial and soul-crushing work. (have to be careful about second-order effects but there is a lot of soul crushing work. which itself raises questions about positive-sumness of human activity)
  - directly optimizing joy and connectedness. have to be careful about this since, eg, Facebook tries to do this and hasn't necessarily succeeded. but 'AI for depression' style research seems useful if it can be done in a real way.
  - what has created joy and connectedness in my life?
    - music: making it easier for people to create / compose / play, especially in groups
    - sports. especially soccer.
    - education, learning things! building a social fabric around
  - learning and teaching (eg the williams CS lab or mathstat library). and relationships of course. travel with friends.
  - AI alignment research. anything that ultimately helps optimize the world for human joy. but this is abstract because you want to be sure your research will be impactful. But I think working at the edges of applied value alignment can be useful.
    - the ideal of this is combined with computational life coaching: a system that learns enough about people's life courses and inner thoughts and happiness to know who they are, what they want, and start making suggestions that would make them happier. including stuff like 'it would be good for you to mentor someone' and then it knows someone else who really needs mentorship, and can maybe jumpstart that relationship.
    - obvious counterpoint is that understanding high-level issues of depression and life happiness is difficult and it's hard to imagine a single agent outperforming the entire field of psychology at least at first. Maybe better to just automate all the hard menial tasks and then let humans sell psychotherapy services to each other?
  - METAPOINT: I don't have to have the answer to every question. It's fine to blog about some thoughts, and admit the ways in which I'm still uncertain.
putting yourself behind your arguments
- When I say "AI safety is an important topic" there's an implicit assumption I mean that people should work on it, and in particular that I should work on it. If I'm not myself working on it, or don't see a path for working on it, then why should anyone be convinced then I say that they should work on it?
- One of the weird things about the "real world" is that people are expected to be able to convert their arguments to action. If I say that it's "possible" to build a model that's immune to adversarial examples, people interpret that as a potential commitment to actually build that model (or at least, a claim that I could build it if asked). When actually I just meant an existence-proof argument: the brain is immune to most forms of adversarial examples we see (like yes, there are illusions, but no one has ever come up with a picture that a non-pressured human brain will say with 99% certainly is a parrot even though it's actually a pineapple), so clearly there's some model that does it; it can't be impossible.
what do I believe that other people don't?
- the importance of exposition in research. obviously lots of poeple believe this but it tends to be lost on actual researchers. they become focused on communicating to their 'tribe' and lose track of others not in the tribe.
- life is not necessarily positive-sum.
- religion is valuable and necessary, its loss is a social crisis. we need new forms of community and identity.
what it means to be a person.
- we build models of ourselves. my future self is a stranger in many ways. I sometimes enjoy things I don't expect, or don't enjoy things I thought I would. Moods change and circumstances change. I do things and want things that are different from what I'd have done or wanted before. (and there is a theory that this self-awareness is what consciousness is: our notion of agency invented to support external social behavior, turned inwards so we can reason about our own selves at higher abstractions).
- so a statement like "I like X" (to play music? or listen to a certain kind of music? do a certain kind of work? a certain sexual preference? criteria in an apartment search?) is not entirely a bottom-up base desire, it's also model-based reasoning, a prediction that your future self will like most of the concrete experiences that the broad class of X-experiences grounds out to. So it's a model both of your future self and of the class of X-experiences.
- These models can be formed and strengthened. Part of growing up and building an identity is understanding things you like and don't like. And this corresponds in part to skills -- having a lot of skill in an area gives confidence you can navigate the general class of X-experiences to create the specific experience you like (also just exercising skill is itself satisfying, as in violin playing). An adult knows who they are and can put themselves into situations where they'll be good at reacting. They are like an RL agent that knows where they've received training data and can act to prevent covariate shift (though this is maybe a slightly different issue).
- but these models can also be broken. if you think you like something, and then experience doing it and not enjoying it, that is a strike against your model. if you do enough of this, the model gets weaker and weaker, until you can make no confident predictions about things you like. this happened to me, partly with backpacking, partly with music, partly with CS research, until there is nothing that I enjoy anymore.
- lessons from this? how to build new models? maybe still be guided by skills (since these are inherent advantages in enjoying a domain) and relationships (good to learn things that bring you closer to people).
how I misinterpreted Teller
- the quote about sometimes magic is just spending an unreasonably large amount of time on something
- I interpreted this as, I should be willing to go deep and work harder than anyone else and put in tons of time
- but this manifested as relying on system 2 rather than system 1. I learned to work slowly and doing anything takes me a ton of time.
national pride is not earned. it's like gay pride
- inspired by this (david brooks column)[https://www.nytimes.com/2018/02/26/opinion/millennials-college-hopeful.html?action=click&pgtype=Homepage&clickSource=story-heading&module=opinion-c-col-left-region®ion=opinion-c-col-left-region&WT.nav=opinion-c-col-left-region]
  - which quotes a millenial as saying she has no pride in the US.
- lots of liberals tend not to have national pride. because the country "doesn't deserve it", has objectively done bad things, continues to do bad things and mistreat our people
- e.g., https://www.theodysseyonline.com/why-im-not-proud-to-be-an-american
- this feels like a person saying, I've screwed up my life, I've done bad things, I've hurt people, I'm ashamed and depressed.
- but the solution to this is always to have pride. pride is not an objective thing, that you have to justify with virtue. pride is the opposite of shame. pride is the belief that you are good at your core, that you deserve to shine
- and that belief is almost always necessary to a recovery
- is there political science evidence for something like this? national pride as correlated with good outcomes? obviously the hard thing would be showing causality in the direction I want, which is the counterintuitive direction
- american nationalism at its best is idealistic nationalism. we are the first country that's tried the experiment of democracy, of integration, of welcoming and economic opportunity.
- every other country is defined by a people. america is defined by ideas. that's worth being proud of
- link to noah smith's posts about idealistic nationalism
- ((honestly I'm not sure I believe this whole post, have to think it out more)
the importance of motivation
- start with a story (when I was a kid, ???)
- thesis: being good at something is not, mostly, about innate talent. it's about caring about the thing. caring enough that you will spend effort learning about it, coming up with your own questions and trying to answer them, self-teaching.
- there's nothing normative about this, no right answer to what you should care about. As a kid I cared about math and programming more than I cared about analyzing high literature, sports, sexual exploration, or being a good friend (I did care about all those things and wasn't terrible at some of them, but there were priorities. if you'd asked me I would have said, of course I care about being a good friend, but in practice I did not spend my nights and weekends obsessing about how to be a better friend). So I spent more time on those things and became relatively more capable at them.
- it's possible to not care about anything particularly much, and then become a person with no particular capabilities. this is the failure mode of a bunch of kids. it is also the failure mode that I seem to be going through, much later in life.
- in grad school I stopped caring about CS research. I had cared about it because I thought I would do stuff that mattered, that would really change the world. Which sounds naive, but tech really has changed the world and the messianic feelings are sometimes justified. But I ended up working on things I hated, that didn't matter, that no one cared about, while simultaneously learning to disrespect much of 'mainstream' ML research. And at the end of it, determining that most research doesn't matter, that the 'useful' stuff is mostly trivial, and the abstract interesting stuff has no impact. And that even advances in tech are probably not good for humanity.
- (on the other hand, it was the only thing I ever really cared about. you know the saying about how 'everyone disbelieves in most gods, athiests just disbelieve in one more'. everyone doesn't care about most things. post grad school I just don't care about one more -- the only thing I ever cared about -- which leads to depression).
- and the effect of no longer caring is that I became worse at it. I wasn't thinking all the time about new and novel research questions. I wasn't excitedly discussing my research with others and building relationships and community. I wasn't actively learning new and relevant material. I wasn't contagiously enthusiastic.
- the main thing grad schools want is motivation. 'highly motivated' is one of the most positive things you can put in a rec letter. motivation causes you to work hard to be good at things. we've made motivation into a normative thing. being motivated is good and admirable, being unmotivated is bad.
- this is especially depressing because of the feedback loop of failure. losing motivation is a cause, in some sense almost the definition of depression. and it makes you worse at things, which is even more unmotivating, and this is a feedback cycle. But this is not just a dispassionate phenomenon, it's a cycle by which you're becoming a worse person by the standards of everyone in academia. People who are not getting good research done are not worth talking to, not worth allocating time for, not worth including in the community. People might try to claim this is not true, but it's hard to hide the fundamental nature of academia in which top people only want to advise strong motivated students, and your continued existence in the community (in the sense of getting a stable long-term job) is contingent on maintaining motivation and contagious excitement about your work. People literally care about you less when you're depressed. And of course you've internalized these same standards, so you care about yourself less, and think you are losing worth. I'm not trying to argue that the system could or should be otherwise -- attention and jobs are scarce resources and of course they need to be allocated intelligently, so there are reasons why things are the way they are -- but even if we are living in the best of all possible worlds, yet that world is still bad.
- all this despite the fact that motivation is not normative. people are motivated about a lot of things that they shouldn't be excited about. everyone in academia knows of other groups or entire research communities doing work that they consider unimportant. and often the feeling is mutual. so we can't be sure which work is not worth being excited about, but we have a nonconstructive proof that at least some work that people are motivated about is bad.
- what do you care about? a lot of it is social and status-driven. if you are friends with a bunch of math people, you'll think math is important. if you're friends with a bunch of AI safety people, you'll think AI safety is important. if you're friends with a bunch of social justice activists, you'll think that's important. etc.
what is AI?
- see argument between Chollet and Pfau: https://twitter.com/fchollet/status/904967157590532096
- AI is either 'building intelligent machines', in which case it includes basically all of CS and programming, the mechanization of all forms of thought and information processing. Or maybe it is 'goal-oriented' behavior, perhaps formulated with the full Bayesian decision theory framework - beliefs, utilities, planning, a full agent. Or it is AGI - something that has the full spectrum of flexible intelligence as a human, as defined by say the turing test (though most would say this is neither necessary nor sufficient).
- Definition A is so broad as to be almost vacuous, every web developer is 'building AI'. But in some sense it's the relevant one. Automation putting people out of work doesn't have to be sexy deep learning magic. Lots of people's jobs can be replaced by a database and a web form. So for automation, defn A is relevant.
- For superintelligence, defn B is relevant. Goal-oriented behavior is dangerous if the agent becomes too good at pursuing its goals. But you can get dangerous superintelligence without necessarily passing the turing test (eg, corporations).
- If, like me, you care about understanding extant human intelligence. Who are we, what makes us tick? How can we build entities that are like us in interesting ways? Then AI cannot be just about goals. Because humans ourselves are not fundamentally goal-driven agents. Goal-driven agents are an interesting and possibly very consequential field of research, but human beings don't have utility functions and fixed goals. We learn goals from our environment, from other people, from those we love and care for. But we ultimately get happiness (/thriving/whatever) from defining and achieving goals. If we were really directed towards a particular goal, we would be unhappy if not making progress towards that goal. But it turns out humans can be happy with any goals, as long as we are making progress towards those goals.
- Evolution gives us a meta-level 'goal' of reproduction. So the human brain as a program is at some level the result of a search over programs trying to maximize reproduction. But that doesn't mean that this program internally reifies the goal of reproduction, or is in any way reducible or explainable as a Bayesian agent.
- How should the field of AI define its goals? Well first of all, AI qua AI is not a useful goal - we should figure out our actual goals (human thriving) and then solve the problems necessary for those goals.
humans do not have final goals
- the AI paradigm assumes a utility or reward function. rational agents are, ultimately, trying to take actions that maximize their total reward (since we are usually uncertain over the effects of our actions, we consider expected reward). this is a final goal. in the process, we compute things like value functions. Getting to states of high value is an instrumental goal. The high-value state has no worth in itself, but it is useful as an intermediate target, as a step in the process.
- Human beings have no true final goals. Sure, all human drives are implicitly derived from the evolutionary goal of reproduction, but actual human brains do not act as reproduction-maximizing agents (this is obvious from the fact that many people choose not to reproduce). But there is no single purpose from which all other human behavior can be derived.
- (I need to flesh this out a lot more:
  - religion provides 'final' goals but few people derive all their behavior directly from religious principles -- we have bundles of competing impulses -- and in any case religions are learned and can be unlearned or changed, whereas true final goals can never change
    - - if you have some overarching principle that allows you to decide one final goal is better than another, than neither of those things were ever final goals in the first place (is vs ought dichotomy)
  - people think they have final goals like 'be happy', from which they derive instrumental goals like 'find good friends and relationships'. but then to other people, those isntrumental goals become final goals. eventually you get far down into the weeds of 'sell more widgets' or 'write snarky tweets' or 'continue republican control of congress' and there exist people who think of these as their final goals without any reference to higher goals they might have originally been instrumental to
- people have limited cognition and can't hold in their heads an entire hierarchy of final and instrumental goals. so you 'run the subroutine' of executing an instrumental goal and, for a while, that becomes your effective final goal. then you have to try to remember, or reconstruct or infer, what the original final goal was. but maybe you come up with a different one, and you never really execute on your original final goal. CONCRETE EXAMPLES OF THIS? you'd think this is bad -- by forgetting the original goal you will fail to achieve it -- but goals are arbitrary so it could be good that you end up with a new goal that is maybe more achievable.
  - moral philosophy is the business of coming up with a final goal from which all human behavior should be derived (eg maximize global utility). Of course this suffers from the is vs ought dichotomy because how can you choose between final goals without some other normative principle? (which itself would be your real final goal). So moral philosophy is doomed.
  - norms like free speech (developed as an instrumental goal, JS Mill argues why it is good) and democracy (ditto) become effective final goals.
  - somehow maybe reference value alignment and people learning goals from others.
academia is a community. you can't do research on your own [copied from 5/2/16 diary].
- why not? motivation is almost impossible. socially-imposed goals and deadlines and objectives give the work meaning.
- also, in line with Thurston's "proof and progress", a theorem proved in a book is meaningless. in a very real sense, the output of research is social understanding, which can only exist in a community.
- it's also a community in the "good ol' boys'" club sense which is problematic in many ways but also almost necessary. like the thing that matters most in academia are not skills that can be easily tested for, but like a sense of identity that is very socially driven, so the only way to really test for it is social proof. (though you don't want to go too far with this, obviously objective metrics are useful too).
- grad school is supposed to train you to be an academic. and part of this is gaining skills. but part of it is joining a research community. which means:
  - as a prospective student, one of the strongest criteria for choosing an advisor is,
    - do they run their research groups like a community, where students feel like they are working together towards a common goal (even if on separate projects)?
      - community among broader grad students at your department but not in your research area is also nice, but not sufficient on its own since your research identity is really what matters for an academic career.
    - are they connected to the (or an) academic community? will they help introduce you to that community? things to check here: collaborations, service on PCs, etc. and even just "what is the community", e.g. there is no real community for PhdAdvisor. This is the problem with idiosyncratic or interdisciplinary thinkers - they are incredibly important, but terrible advisors unless they have succeeded in creating a new community around their intersection of disciplines. Questions of community are hard to judge as an outsider, so you want to ask your own advisors, or even other faculty that you talk to when visiting schools.
  - as a grad student, you should make sure you're engaging socially, contributing to the community of your research group, going out of your way to meet people at conferences, being interested in their research, etc. what else?
  - as an advisor, you need to understand that the single best predictor of grad student productivity and motivation is whether they feel connected, socially validated, that their research is important because other people care about it, that they have people to talk to about their ideas and frustrations. little things matter here, like shared codebases and formal onboarding, having students review each others' papers, lab retreats and 'fun' events, etc. but also big things, like don't assign a student a project that will take them their whole PhD to complete and has no natural opportunities for collaboration. Make sure you encourage younger students to work with and learn from older ones. Etc.
- NOTE: if I write this blog post, it should be maybe structured as a series of short posts so that people will read them. Also obviously I will only actually write it if I still feel the fire on this issue at some point when I actually have time/inclination to write stuff.
models are priors on decision boundaries (on policies?) [copied from 5/2/16 diary].
- totally a technical point. but like use the example of GMMs versus logistic regression. In logistic regression, the weight vector could be anything. But in a GMM, if I show you the X values corresponding to two obvious Gaussian clusters, then you have a pretty strong idea what the separator should be even before you see any labels. So this is a much stronger (and more useful) prior than a Gaussian or whatever.
depression in academia. [copied from 5/2/16 diary]
- it's not just a random thing that sometimes magically happens to grad students. it's not that there's some chemical imbalance in the brain that just happens to manifest in grad students at a much higher rate than the rest of the population. depression is caused, by isolation, lack of a community, lack of support, and the feeling of having wasted years of your life on something that you thought was important but does not seem to have loved you back. (this could connect to the "academia as a community" post series).
  - an analogy: cancer. anyone could be susceptible, but we know a lot of the risk factors. a work environment filled with smoke and carcinogenic chemicals is unhealthy and unsafe. If 50% of your employees had cancer after six years, that would not be considered okay. But somehow 40% of grad students being depressed is considered okay.
modern tech companies as a new form of business that captures your identity. [copied from 5/2/16 diary]
- marxist 'alienation' is all about doing work for hire, doing someone else's work, and it sucks. people are hugely more productive when they are working towards a goal they identify with, that they feel is important and that they will be rewarded for. This is the magic of entrepreneurship, where you are literally working for yourself, but also of startups where everyone shares a vision and a goal. (other examples: elon musk companies, Amazon apparently, but even Google/Facebook/etc). And it is a legitimately empowering and joyful principle to organize human activity around, with a lot of advantages over alienated work-for-hire. BUT.
- it has all the same dangers as academia with burnout and depression and failure. identifying with your work makes it really tough when things aren't working out.
- it is also easy to exploit and fake. the advantages of this culture are so great that companies are incentivized to 'fake' it, to motivate employees to go all-in on something that is really just creating wealth for the owners. this is the worst of all possible worlds.
- people's identities go through natural shifts, sometimes you want to identify with your work, and sometimes it's more important to be a parent or a friend or a caregiver or to focus on hobbies. and it's important to support those roles, either within a company or at least within a society.
- self-actualization is in some ways a point of privilege. It's great to have an identity that is wrapped up in a set of skills that allow you to change the world, but there are a lot of people who still have no good options other than alienated labor and probably won't for a long time. and even if they could have options, actually developing those identities is a social thing which is, again, a point of privilege or a good ol' boys club.
- there's a direct conflict between letting everyone be self-actualized, and using other humans as subroutines to achieve your own goals. if you need something done and you tell someone to do it, they are not doing it for their own purposes and are not going to be super invested in it. this is bad because do to really big things you might need a large organizational hierarchy. solutions seem to include:
  - get people on board with the idea of the hierarchy. if you believe Elon Musk's goals are super important, and trust him to achieve them, you might really be excited to do whatever it is he tells you.
  - not everyone in the hierarchy has to be self-actualized. accept that there will be some people who are just putting in their hours as a standard alienated job. and this is fine because there are lots of people who don't want to wrap their identity in a job for previously discussed reasons. but it's a tricky cultural issue.
the importance of friendly AI. [copied from 5/2/16 diary]
- this might not be the best title. but I do think there's a seriously important point here.
- humans don't have a coherent value function. we derive almost all of our values from social interaction. we have friends who care about rap music, so we care about it too. we have friends who care about helping the homeless, or proving theorems, or getting rich, or praising God, so we care about those things too. Our entire lives are quests for connection, social validation, social status, and importantly meaning. Notions of a "human utility function" as some platonic object that would be great if we could convey to robots, and it's just unfortunate that we can't write it down directly so we have to learn it instead - those are misguided to a certain extent (cf utilitarianism). So much of human life is the search for meaning, the search for motivation, for things to care about. And the answers we find end up being very much social; we get meaning from friends and communities that represent ideas and identities we find valuable, but also the communities are just important for their own sakes.
- so an AI that wants to operate within human society, e.g., pass a turing test, is going to need to reflect this. it's going to need to want friends, to learn its values from others, and to be genuinely curious about what is important for it to be doing.
- this has the nice side benefit that it solves many issues of safety, and the 'off-switch'. (cite dylan's work). but it's also just a fundamental design consideration for AIs. you'd much rather buy a household robot, or a chatbot, or whatever that is genuinely interested at a design level in figuring out what you want and helping you achieve it, than one that is just "faking it" with a preprogrammed utility function.
- questions, obviously.
  - is this a meaningful distinction? sure, humans sometimes get lost in their search for meaning. doesn't this just mean that their real utility function is "find meaning" and we just have trouble optimizing it?
    - answer: that utility function is ill-defined at best, and the implicit defintions that it seems like people end up with are essentially driven by social notions of meaning.
    - also: uncertainty over utility functions is fundamentally different because it creates a value of information? though couldn't there be another utility function defined over an extended space that incorporates the induced value of information? but now this is a different space so it's not directly comparable.
    - also: there is no natural correspondence between simple utility functions and simple implementations. simple utility functions can be tremendously complicated to optimize. whereas very simple procedures can turn out to optimize tremendously complicated utility functions (examples of this? connections to models and inference, simple models requiring complicated inference, whereas simple frequentist procedures sometimes correspond to weird/complicated bayesian models). it may be that human behavior is implicitly optimizing some really complicated utility function (in a way that is nonobvious to us, like people in 1700 were wrong about slavery, but "the arc of history bends towards justice") but that the behavior of seeking socialization is itself the most compact specification of the ultimate utility we care about. (NOTE: this still feels like probable bullshit, would need to flesh it out before committting to it, in some sense humans are really just optimizing reproduction and everything else just falls out of that…).
most arguments people have are pointless
- someone who 'does everything wrong' can succeed. and often does.
- arguments of the form 'X is better than Y' are often true, all else equal, but don't consider hidden costs or tradeoffs. (they are true but wrong)
- the assumptions underlying the conceptual system in which the argument rests might have changed since the original evidence appeared.
- many arguments are heuristics that people have for making decisions when they don't actually understand the issue. and many complex issues are impossible to understand so following broadly useful heuristics ("be honest", "respect people", "no global variables", etc) is a good strategy. but many arguments consist of dueling heuristics. and the actual truth may be buried somewhere. (arguments can't be trusted)
- could get at this with random game trees? or not even adversarial, just random search problems? the 'true' answer to an argument is the solution to the search problem and heuristics disagree on how to find it.
- in writing this I'm thinking of Vicarious who is ignoring all of AI's conventional wisdom. Or deep learning which also did this. here they had a kernel of certainty which just washed out all the noise from everyone else
- usually the correct answer isn't even within the conceptual realm of what we're considering. like you can't argue your way into deep learning if you're an SVM expert because you've spent too much time gaining expertise to throw it all away. and your view of the world literally doesn't let you see the attraction of doing so.

blog posts to write

Links to this note

AI reflections master

kernel

current identity, goals, and plans

Meta