'[ no free lunch theorem ]' arguments are misleading because they consider the space of all possible functions. In fact, we usually care…
NameRedacted points out the strong free will theorem . This says that electrons sometimes have 'choices': situations where their behavior…
[ my goals ]
Should I give up having fun in order to do impactful things? It'd be tempting to think that it'd be more virtuous to be serious and mission…
There's often a lot of space between learning [ fundamental ]s and being able to do a thing. Understanding Turing machines didn't…
Examples recommended by GPT4: Long Short-Term Memory ([ LSTM ]) : Paper: "Long Short-Term Memory" by Sepp Hochreiter and Jürgen Schmidhuber…
Why would you be 'proud' of something you had no control over? The core revelation for me was that pride is the opposite of shame . Most…
Is there such a thing as 'general intelligence'? What capabilities does it require? Is it a goal worth striving for? We usually speak about…
Rules that work in many situations are valuable. If you can cook a burger, you're a McDonald's employee. If you can specify the rules for…
If you need to open a specific lock, you can use a key that encodes the precise information needed to open that lock. If you need to open…
Fundamentally, where does generalization come from? [ causality ]: a model may generalize because it has discovered the true mechanism, or…
Sutton and Barto use this as a general term for any form of interleaving policy evaluation steps with policy improvement steps. This…
Many objects can be generated by a sequence of actions. For example: Generating language by adding one word at a time Generating a molecule…
Small talk can be mindnumbing and pointless. I like the idea of 'big talk, not small talk'. But realistically small talk serves a social…
"What I cannot create, I do not understand". Related to: [ computational complexity ]: provers vs verifiers. [ P != NP ] [ production vs…
a Buddhist point. I was trying to fast last night and kept being tempted to relax it slightly . "It won't hurt anything if I just have one…
An intelligent [ agent ] should work to understand the world. This understanding takes the form of a set of relevant [ abstraction ]s, a…
Chemically, glutamate is the [ acid-base chemistry|conjugate base ] of glutamic acid (an [ amino acid ]). It is an [ anion ]; its sodium…
If you fail to achieve your goals, you'll be sad---almost self-evidently. Nonetheless: there is no coherent notion of the 'right' goals to…
Visit NameRedacted Visit SuccessfulFriend Visit Asian cities Do real bulking and cutting to get in shape Move Learn Mandarin Learn…
see also [ thoughts about kids ] Get them reading young---ideally by age 2 (like NameRedacted). Learn an instrument from a young age. Let…
One can never fully deserve grace, but may receive (and extend) it regardless. Probably the most beautiful conceptual contribution of…
Thoughts drawn from my experience doing a CS PhD at a top-4 school around 2010-2016. They may be somewhat applicable to PhDs in other areas…
A recent survey found that ??% of Berkeley grad students suffer from depression. This should be shocking and dismaying. Yet no one seems…
Why do we clip gradients in deep learning? When is it important and what is the right way to do it? It seems like the standard recipe used…
For a normalized distribution , constructed from an (unnormalized) energy with normalizing constant as a function of parameters , in…
How should I think about grades when [ teaching ] a class? I believe in mastery learning. Feedback isn't useful unless students have the…
"How did you go bankrupt? Two ways. Gradually, then suddenly." - Hemingway. It can take a long time to lay the foundations for significant…
A 'graph neural net' is a differentiable, parameterized function whose input or output (or both) is a graph. Discriminative: graph as input…
see also: [ great shows ] Cloud Atlas Pulp Fiction
see also: [ great movies ] Deadwood Sense8
The greek letters used most commonly in finance are probably alpha and beta from the [ single-index model ]. However, the term 'greeks…
When someone we love dies, a sense of possibility leaves the world. Our grief is usually proportional to how much we cared about them…
A nice observation from Percy Liang on the relationship between language modeling and grounded understanding: Just because you don't…
(related: [ communication is processing ]) A big part of growing up is communicating to your future self. Your future self isn't going to…
A thought that just occurred watching Billions. The two deputy attorneys (white guy and black woman ??) are in her apartment, which is nice…
Opposite of a fixed or 'scarcity' mindset. It's important to recognize that the world is nonzero-sum and that improvements are possible. We…
these include things like: posing interesting questions for discussion useful [ generative questions ] like: what have you been reading…
At ~33 my hairline has been receding and is starting to thin in front. I have a few options: Accept hair loss and go bald. Accept hair loss…
A New Yorker article on [ happiness ]: http://www.newyorker.com/tech/elements/a-better-kind-of-happiness discusses happiness as a source…
Closely related to [ discrete latent variable ]s and to [ reinforcement learning ] with discrete actions. If I do a thing and it goes well…
On Michael Taft's podcast, A. H. Almaas pointed out that an obstacle for most people realizing a sense of divine, nondual love, is some…
Pretend that some other group has published the paper you're imagining: are you excited to read it? Write down ten ideas and ask a mentor to…
The plans I make now include components that would have been impossible for me to conceive of as a kid. At the moment (July 2020), I'm…
What are some things that people are doing now that are just clearly valuable? 3blue1brown : So much of math is hidden behind notation…
[ Dan Brown ] likes to say that 'the view is the meditation'. That is, meditation isn't
People on Reddit worry that there are hundreds of new ML papers every day---how could you possibly keep up? How can you filter the firehose…
[ be open to friendship ] [ act as if you're already friends ] [ people like hearing their name ]
campfires apres ski choir rehearsing under a crisp fall night sky ragnars in vans shabbat various gay gatherings (e.g. pride) -- here the…
Breville/Polyscience Control Freak: induction burner with temperature sensor. replaces normal stove, electric kettle, maybe sous vide, maybe…
from 2017: remember things I am at least somewhat an expert in, that other people may not have seen: philosophy of bayesian statistics…
I'm sometimes tempted to look back and find patterns in my life, and identify those as "who I really am". For example: maybe I want to be a…
(originally written as a Google doc between 2010-2012) most of this advice is obvious, but still good to remember. students appreciate food…
Multiple senses: An 'ill-conditioned matrix' has a large ratio between its largest and smallest eigenvalue (more generally, see what is a…
References: Gu et al., Continuous Deep Q-Learning with Model-based Acceleration (2016). A technique used in [ model-based rl ], where we…
The Eliezer Yudkowsky school of thought is that immortality is possible, and obviously desirable; any other position is [ learned…
One aspect of depression recently has been feeling like things are pointless, there's nothing valuable for me to do. My entire PhD has been…
[ Shinzen Young ] reframes the traditional Buddhist concept of impermanence as "flow". I think this is starting to make sense to me. I…
Examples: SGD prefers some minima over others
Importance sampling allows us to compute expectations under a distribution using samples from a different distribution , by weighting the…
[ grokking ] / [ phase change hypothesis ] emergence of near-discrete features in large transformers symmetries / non-[ identifiable…
For me, imposter syndrome felt like knowing that there was something deeply wrong with me, something I was missing or didn't get, that would…
In the 21st century, humanity is developing a new form of engineering. Rather than manually designing artifacts, we are optimizing over…
Indole alkaloid - Wikipedia Indole - Wikipedia A benzene ring fused with a [ pyrrole ring ]
Ways to specify inductive bias: Feature engineering Prior distribution acts as regularizer in MAP estimates Graphical model (constraint on…
Talia Ringer says that these are one of the most beautiful, foundational ideas in programming languages: https://twitter.com/TaliaRinger…
Around 15min into https://www.youtube.com/watch?v=gWAZFuz_mFc , [ Ram Dass ] claims there are 'infinite doorways'. You start on the…
The Leibniz calculus notation using infinitestimal quantities like or is simultaneously Very sensible and intuitive, but also Constantly…
References: Bae et al. (2022) If Influence Functions are the Answer, Then What is the Question? https://arxiv.org/abs/2209.05364 Grosse et…
To achieve final goals, we have to break them down into a hierarchy of instrumental goals, and then get to work on achieving those. And for…
The [ front-door adjustment ] allows identifying causal affects using a mediating variable that sits on the causal chain between X and Y…
I want to care about an intellectual topic and have friends and colleagues with whom I can enjoy discussing that topic. I used to have…
[ theory of intelligence ]
Boaz Barak writes in GPT as an "Intelligence Forklift." that [ language model ]s seem to function effectively as [ tool AI ] that can…
A lot of discussion around [ artificial intelligence ] implicitly conflates intelligence with [ consciousness ]. It assumes that as we…
Inspired by this tweet: https://twitter.com/davmre/status/841803926051549184 Ways we currently equate intelligence with moral worth: animal…
It's a bit counterintuitive that high interest rates prevent inflation. After all, doesn't a high interest rate mean that the [ central bank…
interesting things dennett says in "darwin's dangerous idea" the "baldwin effect" uses reinforcement learning to construct a version of…
Interfaces enable modularity. In general, standardizing an interface can yield quadratic benefit at linear cost. Suppose we have people…
Things to pay attention to in setting up a new place: Lighting makes a huge difference and can totally change the ambience of a space. Don…
Fasting is a powerful, life-changing idea because it's simple, clear, easy to follow. A diet plan that involves counting calories requires…
References: https://en.wikipedia.org/wiki/Intermolecular_force and linked pages conversations with GPT-4 A molecule is a set of atoms…
The theory of musical intervals almost makes mathematical sense (but not quite): The sensible part: integer ratios The ancient Greeks…
objectives: maximize entropy of the state visitation distribution requires empirical estimates of [ entropy ] maximize mutual information…
David Hume pointed out that there's no logical way to get from 'is' (descriptive) statements to 'ought' (normative) statements. This is…
The isoperimetric problem : among all closed curves in the plane with equal perimeter, which encloses the largest volume? It's well-known…
It can be very hard to hold onto a positive self-image and an [ optimism|optimistic ] worldview, even if you intellectually 'know' these to…
Admitting things that I personally don't know is hard, because it feels like admitting a weakness or failing. But ignorance isn't a…
Apparently NameRedacted told this story in a talk. Sharon Salzberg says that if she could put one thing on her tombstone it would be this…
Notes copied from Google Docs https://docs.google.com/document/d/1G7Gxo-A3gQrlUx3G4BYmH-GjUWbB7eTOJZcuAs2ii0A/edit most of these things…
From 2017: stay publicly active. work on open side projects, and/or publish, and/or blog. live at such a means that I could lose my job and…
Something I've struggled with: given my position of immense privilege, how can I justify doing wasteful 'fun' things like skiing or clothes…
I don't know if this makes sense, but one intuition I have for karma comes from the observation that the weights of a least-squares linear…
We are given the opportunity to bet some fraction of our wealth on a coin flip with probability . We can repeat this as many times as we…
multiple senses: in machine learning: positive definite (Mercer) kernels in linear algebra: kernel (nullspace) of a linear map in CS systems…
Effects in the brain Ketamine is an antagonist of [ NMDA receptor ]s, and activates [ AMPA receptor ]s. References: Ketamine – Lorien…
pg says: the random things that you learn as a kid make you into a key. Your job is then to find the lock that you fit into. But that's not…
Adults tend to talk to and work with other adults. We don't spend nearly enough time addressing the problems of younger people: college…
I just coined the phrase 'kitchen sink' deep learning for a vague idea that comes to me occasionally. Roughly: rather than using a uniform…
An annoying thing about language-learning tutorials is that they often focus on language that you'll never actually use as a tourist. When I…
Taco Cohen speculates on Large Control Policies as a successor to large language models: https://twitter.com/TacoCohen/status…
Much of statistical practice is concerned with distinguishing signal from noise. For example, significance tests quantify the likelihood…
If you believe that neural nets basically just memorize the training data, then training larger and larger models is hopeless. The…
I don't want to be led. I want to be creative and do things that are dramatically new. Telling other people what to do feels almost evil to…
Happens all the time, at small scales and large scales. Large scale: When I was young, it was possible to imagine becoming a confident…
The set of good approaches is often hidden in an exponentially large space. Learning that an approach is not good doesn't help narrow that…
There's no easy or fast solution for feeling good about leaving academia, because you're giving up some aspect of your [ identity ], and…
(related: [ growing up means becoming wrong ], ask for evidence , [ Seeing Like A State ]) Related to some interactions with NameRedacted…
A thing that's tough about going through big personal changes is that it takes a while for your self-model to catch up with your actual self…
People on the internet have very different standards as to when and how it's okay to ask a question. There are roughly two camps: Search…
https://maskofreason.files.wordpress.com/2011/02/the-library-of-babel-by-jorge-luis-borges.pdf Insights illustrated by this story: Naming…
In principle we can apply [ automatic differentiation ] through any composition of differentiable operations. This lets us get gradients of…
I want to change the world. What does that mean? Suppose I create a billion-dollar company. That's an enormous amount of value. It's many…
tags: created: 2023-12-07 modified: 2023-12-07 References: https://arxiv.org/abs/2006.16236 The usual [ transformer ] [ attention…
A linear time-invariant system is one where the dependence of the output on the input is: linear: an input produces an output , and…
podcasts and audiobooks Alan Watts 'out of your mind' lectures
I've heard it said, and it's been ringing true to me, that the thing to do is live the life you want to have now , not plan to spend years…
Some things are genuinely hard to do. But many others I don't do just out of laziness, or maybe lack of [ agency ]. I know that they're…
A atomic orbital with two electrons both attached to the same atom. In contrast to a bond, where each atom contributes one electron. Lone…
Notes on https://www.pragmatic.ml/a-survey-of-methods-for-incorporating-long-term-context/ 'Standard' transformers have O(n**2) complexity…
There's a tendency to focus on things that we have the (conceptual/mathematical/societal) tools to understand, even when we know this is…
How do I justify being only 'mostly' [ vegetarian ]? I know that cows and chickens are abused to produce milk and eggs. Why is avoiding…
What does it mean to love someone? Of course this question has as many answers as there are people, and probably more. But here's one view…
I saw this phrase on Twitter somewhere and it really resonates as a description of the ideal approach to science. There is no real…
A macrostate in statistical mechanics is a collection of base-level states; equivalently, a subset of [ phase space ]. It's what you see…
Mahamudra means the 'great seal' or 'great gesture'. We take and [ hold the view ] that each event arising in [ awareness ] --- every sight…
Managing for high-variance / creative work versus low-variance consistent work: https://blog.sbensu.com/posts/2023-01-18-high-variance…
There are a lot of difficult decisions to be made in life. Maybe you need to decide the business strategy of a company, knowing that good…
An idea I got from [ John Higgs ]'s discussion of metamodernism is that taking [ all models are wrong ] to its logical conclusion requires…
Sometimes I've been scared of losing my identity. In particular I worry about working a non-research job, or having sex with (or being…
I have a private theory about what marijuana does. I'll try to articulate it here. I don't know much about the public theories, so maybe…
A martingale is any [ stochastic process ] that stays the same in expectation. Formally, is a martingale if This condition is related to…
How to think about giving a good massage? Know which way the muscle fibers go. For deep release, exert force perpendicular to the muscle…
Reviewing this 3blue1brown video: https://www.youtube.com/watch?v=O85OWBJ2ayo The matrix exponential is written as E to the power of a…
The Woodbury-Morrison-Sherman matrix inversion lemma, is sometimes useful just for algebraic simplifications. In cases where and are…
Notation for Matrix Multiplication Let and . Then just by the definition of matrix multiplication (the summation over is performing…
References: Hu, Yang (2022) Feature Learning in Infinite-Width Neural Networks https://arxiv.org/abs/2011.14522 Yang, Hu et al. (202…
For any reward function and policy , consider the entropy-regularized reward Taking as our objective the (expected, discounted…
Note: these are personal notes, taken as I was refreshing myself on this material. They're mostly stream of consciousness and probably not…
A function is measurable with respect to [ sigma-algebra ]s on its domain and on its range if the pre-image of any event is…
The core insight that got me interested: "moments of recognizing your thoughts drifting and bringing them back to your breath" are not…
Dec 13, 2023 Sangha session with Dustin: Something I realized during a concentration practice is that being honest about how ‘well’ the…
Generally I think the dharma is deeply true and that [ meditation ] done right is healthy and potentially very beneficial. But I struggle…
Specific states or abilities that can arise from skillful meditation: feeling [ equanimity ] first cessation seeing nimitta accessing…
Suppose we want to do [ automatic differentiation ] on a [ computational graph ] of sequential length . This could equally well be a…
One last thought mental models are so, so important. When I think about computer modeling. It's actually great computers are powerful they…
Like democracy , meritocracy is the worst form of social organization, except for all the others that have been tried. Of course it is good…
References: Risks from Learned Optimization in Advanced Machine Learning Systems A [ reinforcement learning ] algorithm attempts to find the…
Chemically, a substituted [ phenethylamine ]. Like [ dopamine ] but with methyl groups hanging from the two oxygens, and another oxygen…
Generally this means training some aspect of the learning procedure itself. There is then an inner-loop learning procedure, which follows…
Unlike most modern [ deep learning ] systems, humans: don't have separate training/test phases (though we may have wake/[ sleep ]) don't…
Stuart Russell told the story of giving a talk on meta-reasoning at Stanford, with Don Knuth in the audience, where he opened with a slide…
n-methyl-[ amphetamine ]