(note: this is dancing around the issues around why I think [ probabilistic programming is not AI research ], even if it will be a…
Modified: February 10, 2022.
Modified: .
Modified: .
Quit Your Job (palladiummag.com) : Productive exploration requires the application of skilled personal judgment to chasing hunches and…
Modified: February 17, 2022.
Modified: .
Exponential Families, Conjugacy, Convexity, and Variational Inference Any parameterized family of probability densities that can be written…
Modified: May 21, 2022.
Considering training an [ autoregressive ] model of sequence data (text, audio, action sequences in [ reinforcement learning ], etc.), which…
Modified: October 13, 2022.
This note is a scratchpad for investigating the expressivity of the [ transformer ] architecture. In general, one set of intuitions that we…
Modified: October 28, 2022.
Modified: .
In [ Dan Brown ]'s telling, these are: mindfulness ( sati ( https://en.wikipedia.org/wiki/Mindfulness (Buddhism))_): paying attention to the…
Modified: June 27, 2022.
Modified: January 09, 2021.
I grew up in the 2000s reading the New Atheists, where 'faith' was considered a dirty word. Faith was the opposite of reason; it meant…
Modified: November 14, 2022.
MACARONI AND CHEESE 4 Tb margarine 5 tb flour Milk 2 tsp mustard 1 garlic clove, minced or 1 tb prepared minced garlic 2 ½ cups sharp…
Modified: February 22, 2022.
Ultimately what is attractive in fashion is confidence. you can break almost all of the rules if it's clear that you're doing it as a matter…
Modified: July 24, 2020.
I've thought before that fashion is bad because it's about arbitrary trends. But you can also see fashion as good because it's about…
Modified: February 07, 2022.
On an evolutionary timescale, it's useful to evolve structures that can learn quickly. The nervous system is an evolved organ system for…
Modified: October 27, 2022.
See also Scott Alexander's Ontology Of Psychiatric Conditions: Dynamical Systems - Astral Codex Ten (substack.com) theory of depression.
Modified: June 07, 2021.
Modified: July 17, 2020.
A filtration is defined by monotonically increasing subsets of a [ probability space ]; that is, subsets such that we have for all…
Modified: August 27, 2022.
Yale course: https://oyc.yale.edu/economics/econ-251 MIT course: https://www.youtube.com/playlist?list=PLUl4u3cNGP63B2lDhyKOsImI7FjCf6eDW…
Modified: August 28, 2020.
From a Jean Yang tweet : I once attended a talk by the late Nobel Laureate Oliver Smithies where he talked about going into lab on weekends…
Modified: August 21, 2020.
Modified: .
Buddists identify five factors as obstacles to [ concentration ] in [ meditation ]: Sensory desire ( kāmacchanda ) Aversion or ill will (…
Modified: March 22, 2023.
Interesting and seemingly very powerful perspective on the [ cessation of suffering ]. Most refs on this page are from this twitter thread…
Modified: July 14, 2023.
We say that is a fixed point of an update rule if . Update rules can often (though not necessarily) be seen as defining an…
Modified: August 13, 2022.
As AGW points out here , it is statistically better to fit a flexible model family, with an inductive bias, than a constrained model family…
Modified: February 15, 2022.
Modified: .
Credit to NameRedacted for this refrain https://twitter.com/visakanv/status/1324978566455468035/retweets/with_comments It's a powerful take…
Modified: April 13, 2024.
Statements from Forer's experiment : You have a great need for other people to like and admire you. You have a tendency to be critical of…
Modified: July 31, 2023.
A key insight of Christianity is that forgiveness is something we do for ourselves : it's not just about extending [ grace ] to the party…
Modified: April 10, 2024.
Modified: .
These are the first teaching of the Buddha, after he achieved [ enlightenment ] while [ meditation|meditating ] under a tree. The truths are…
Modified: February 25, 2022.
Banks create money by lending. Few understand this. Alice and Bob are on a desert island. Alice has $100, which she deposits in the Desert…
Modified: February 09, 2022.
Some drugs, like cocaine, or DMT, come in multiple forms: as some sort of [ salt ] or as a 'free base'. What's the difference between these…
Modified: January 22, 2022.
Modified: .
'[ no free lunch theorem ]' arguments are misleading because they consider the space of all possible functions. In fact, we usually care…
Modified: December 14, 2022.
[ Andre ] points out the strong free will theorem . This says that electrons sometimes have 'choices': situations where their behavior is…
Modified: October 04, 2021.
Modified: February 10, 2022.
Modified: August 06, 2021.
Modified: August 03, 2023.
[ my goals ]
Modified: September 16, 2021.
Should I give up having fun in order to do impactful things? It'd be tempting to think that it'd be more virtuous to be serious and mission…
Modified: April 04, 2022.
Modified: February 10, 2022.
There's often a lot of space between learning [ fundamental ]s and being able to do a thing. Understanding Turing machines didn't…
Modified: February 25, 2022.
Modified: .
sources: https://en.wikipedia.org/wiki/Gabapentin https://en.wikipedia.org/wiki/Gabapentinoid https://en.wikipedia.org/wiki/Pregabalin https…
Modified: October 01, 2024.
Examples recommended by GPT4: Long Short-Term Memory ([ LSTM ]) : Paper: "Long Short-Term Memory" by Sepp Hochreiter and Jürgen Schmidhuber…
Modified: December 20, 2023.
References: Dauphin et al. 2017 https://arxiv.org/abs/1612.08083 Shazeer 2020 https://arxiv.org/abs/2002.05202 https://arxiv.org/abs/240…
Modified: March 03, 2024.
Modified: July 17, 2020.
Why would you be 'proud' of something you had no control over? The core revelation for me was that pride is the opposite of shame . Most…
Modified: January 20, 2022.
Is there such a thing as 'general intelligence'? What capabilities does it require? Is it a goal worth striving for? We usually speak about…
Modified: August 30, 2023.
Modified: .
Rules that work in many situations are valuable. If you can cook a burger, you're a McDonald's employee. If you can specify the rules for…
Modified: April 28, 2022.
If you need to open a specific lock, you can use a key that encodes the precise information needed to open that lock. If you need to open…
Modified: September 01, 2023.
Fundamentally, where does generalization come from? [ causality ]: a model may generalize because it has discovered the true mechanism, or…
Modified: October 02, 2021.
Sutton and Barto use this as a general term for any form of interleaving policy evaluation steps with policy improvement steps. This…
Modified: March 22, 2022.
Many objects can be generated by a sequence of actions. For example: Generating language by adding one word at a time Generating a molecule…
Modified: March 13, 2022.
Small talk can be mindnumbing and pointless. I like the idea of 'big talk, not small talk'. But realistically small talk serves a social…
Modified: February 25, 2022.
"What I cannot create, I do not understand". Related to: [ computational complexity ]: provers vs verifiers. [ P != NP ] [ production vs…
Modified: April 27, 2020.
Notes on Scott Garrabrant's sequence: https://www.lesswrong.com/s/4hmf7rdfuXDJkxhfg The geometric integral is to products what the…
Modified: April 03, 2025.
a Buddhist point. I was trying to fast last night and kept being tempted to relax it slightly . "It won't hurt anything if I just have one…
Modified: July 19, 2020.
An intelligent [ agent ] should work to understand the world. This understanding takes the form of a set of relevant [ abstraction ]s, a…
Modified: May 22, 2021.
Modified: .
Modified: May 21, 2024.
Chemically, glutamate is the [ acid-base chemistry|conjugate base ] of glutamic acid (an [ amino acid ]). It is an [ anion ]; its sodium…
Modified: August 08, 2021.
If you fail to achieve your goals, you'll be sad---almost self-evidently. Nonetheless: there is no coherent notion of the 'right' goals to…
Modified: July 14, 2023.
Our core notion of a rational [ agent ] is an entity that behaves in a coherent way to pursue some goal. But at a fundamental level there…
Modified: December 05, 2025.
Visit [ Andre ] Visit SuccessfulFriend Visit Asian cities Do real bulking and cutting to get in shape Move Learn Mandarin Learn statistical…
Modified: March 14, 2021.
see also [ thoughts about kids ] Get them reading young---ideally by age 2 (like Amitav). Learn an instrument from a young age. Let them…
Modified: February 25, 2022.
One can never fully deserve grace, but may receive (and extend) it regardless. Probably the most beautiful conceptual contribution of…
Modified: September 11, 2021.
Thoughts drawn from my experience doing a CS PhD at a top-4 school around 2010-2016. They may be somewhat applicable to PhDs in other areas…
Modified: February 25, 2022.
A recent survey found that ??% of Berkeley grad students suffer from depression. This should be shocking and dismaying. Yet no one seems…
Modified: March 06, 2022.
Why do we clip gradients in deep learning? When is it important and what is the right way to do it? It seems like the standard recipe used…
Modified: April 29, 2023.
Modified: .
For a normalized distribution , constructed from an (unnormalized) energy with normalizing constant as a function of parameters , in…
Modified: July 09, 2022.
How should I think about grades when [ teaching ] a class? I believe in mastery learning. Feedback isn't useful unless students have the…
Modified: December 19, 2020.
"How did you go bankrupt? Two ways. Gradually, then suddenly." - Hemingway. It can take a long time to lay the foundations for significant…
Modified: February 22, 2022.
A 'graph neural net' is a differentiable, parameterized function whose input or output (or both) is a graph. Discriminative: graph as input…
Modified: June 06, 2020.
see also: [ great shows ] Cloud Atlas Pulp Fiction
Modified: April 04, 2022.
see also: [ great movies ] Deadwood Sense8
Modified: April 04, 2022.
The greek letters used most commonly in finance are probably alpha and beta from the [ single-index model ]. However, the term 'greeks…
Modified: November 30, 2022.
When someone we love dies, a sense of possibility leaves the world. Our grief is usually proportional to how much we cared about them…
Modified: August 21, 2020.
Modified: .
A nice observation from Percy Liang on the relationship between language modeling and grounded understanding: Just because you don't…
Modified: April 29, 2022.
(related: [ communication is processing ]) A big part of growing up is communicating to your future self. Your future self isn't going to…
Modified: April 05, 2020.
A thought that just occurred watching Billions. The two deputy attorneys (white guy and black woman ??) are in her apartment, which is nice…
Modified: February 25, 2022.
Opposite of a fixed or 'scarcity' mindset. It's important to recognize that the world is nonzero-sum and that improvements are possible. We…
Modified: January 09, 2021.
these include things like: posing interesting questions for discussion useful [ generative questions ] like: what have you been reading…
Modified: February 25, 2022.
Modified: May 14, 2021.
Modified: .
A New Yorker article on [ happiness ]: http://www.newyorker.com/tech/elements/a-better-kind-of-happiness discusses happiness as a source…
Modified: February 12, 2022.
Closely related to [ discrete latent variable ]s and to [ reinforcement learning ] with discrete actions. If I do a thing and it goes well…
Modified: January 23, 2022.
On Michael Taft's podcast, A. H. Almaas pointed out that an obstacle for most people realizing a sense of divine, nondual love, is some…
Modified: November 23, 2023.
Modified: .
Modified: .
Pretend that some other group has published the paper you're imagining: are you excited to read it? Write down ten ideas and ask a mentor to…
Modified: January 09, 2021.
Modified: .
Modified: .
Modified: .
Modified: March 16, 2022.
The plans I make now include components that would have been impossible for me to conceive of as a kid. At the moment (July 2020), I'm…
Modified: July 08, 2020.
What are some things that people are doing now that are just clearly valuable? 3blue1brown : So much of math is hidden behind notation…
Modified: April 02, 2021.
[ Dan Brown ] likes to say that 'the view is the meditation'. That is, meditation isn't
Modified: February 10, 2022.
People on Reddit worry that there are hundreds of new ML papers every day---how could you possibly keep up? How can you filter the firehose…
Modified: December 08, 2021.
[ be open to friendship ] [ act as if you're already friends ] [ people like hearing their name ]
Modified: March 02, 2022.
campfires apres ski choir rehearsing under a crisp fall night sky ragnars in vans shabbat various gay gatherings (e.g. pride) -- here the…
Modified: July 13, 2020.
[ value alignment ] research often frames the problem as: first, learn the human 'value function' --- for every possible state of the world…
Modified: June 17, 2024.
Modified: .
Breville/Polyscience Control Freak: induction burner with temperature sensor. replaces normal stove, electric kettle, maybe sous vide, maybe…
Modified: November 02, 2021.
Modified: .
Modified: July 19, 2020.
from 2017: remember things I am at least somewhat an expert in, that other people may not have seen: philosophy of bayesian statistics…
Modified: February 15, 2022.
I'm sometimes tempted to look back and find patterns in my life, and identify those as "who I really am". For example: maybe I want to be a…
Modified: February 22, 2022.
(originally written as a Google doc between 2010-2012) most of this advice is obvious, but still good to remember. students appreciate food…
Modified: February 07, 2022.
Multiple senses: An 'ill-conditioned matrix' has a large ratio between its largest and smallest eigenvalue (more generally, see what is a…
Modified: December 28, 2022.
References: Gu et al., Continuous Deep Q-Learning with Model-based Acceleration (2016). A technique used in [ model-based rl ], where we…
Modified: July 19, 2022.
Modified: .
The Eliezer Yudkowsky school of thought is that immortality is possible, and obviously desirable; any other position is [ learned…
Modified: January 24, 2022.
One aspect of depression recently has been feeling like things are pointless, there's nothing valuable for me to do. My entire PhD has been…
Modified: February 10, 2022.
[ Shinzen Young ] reframes the traditional Buddhist concept of impermanence as "flow". I think this is starting to make sense to me. I…
Modified: .
Examples: SGD prefers some minima over others
Modified: March 02, 2022.
Importance sampling allows us to compute expectations under a distribution using samples from a different distribution , by weighting the…
Modified: July 05, 2022.
[ grokking ] / [ phase change hypothesis ] emergence of near-discrete features in large transformers symmetries / non-[ identifiable…
Modified: September 14, 2022.
For me, imposter syndrome felt like knowing that there was something deeply wrong with me, something I was missing or didn't get, that would…
Modified: February 10, 2022.
things I learned in [ improv ] class there's no need to try to be clever or funny. if you're present and responsive, trust that the…
Modified: September 13, 2024.
In the 21st century, humanity is developing a new form of engineering. Rather than manually designing artifacts, we are optimizing over…
Modified: February 25, 2022.
Modified: .
Indole alkaloid - Wikipedia Indole - Wikipedia A benzene ring fused with a [ pyrrole ring ]
Modified: May 14, 2021.
Ways to specify inductive bias: Feature engineering Prior distribution acts as regularizer in MAP estimates Graphical model (constraint on…
Modified: January 16, 2022.
Talia Ringer says that these are one of the most beautiful, foundational ideas in programming languages: https://twitter.com/TaliaRinger…
Modified: January 13, 2022.
Around 15min into https://www.youtube.com/watch?v=gWAZFuz_mFc , [ Ram Dass ] claims there are 'infinite doorways'. You start on the…
Modified: October 03, 2021.
The Leibniz calculus notation using infinitestimal quantities like or is simultaneously Very sensible and intuitive, but also Constantly…
Modified: September 06, 2022.
References: Bae et al. (2022) If Influence Functions are the Answer, Then What is the Question? https://arxiv.org/abs/2209.05364 Grosse et…
Modified: January 03, 2024.
Modified: August 14, 2021.
To achieve final goals, we have to break them down into a hierarchy of instrumental goals, and then get to work on achieving those. And for…
Modified: February 25, 2022.
The [ front-door adjustment ] allows identifying causal affects using a mediating variable that sits on the causal chain between X and Y…
Modified: February 25, 2022.
I want to care about an intellectual topic and have friends and colleagues with whom I can enjoy discussing that topic. I used to have…
Modified: November 15, 2019.
[ theory of intelligence ]
Modified: May 14, 2021.
Boaz Barak writes in GPT as an "Intelligence Forklift." that [ language model ]s seem to function effectively as [ tool AI ] that can…
Modified: September 29, 2023.
A lot of discussion around [ artificial intelligence ] implicitly conflates intelligence with [ consciousness ]. It assumes that as we…
Modified: January 18, 2023.
Inspired by this tweet: https://twitter.com/davmre/status/841803926051549184 Ways we currently equate intelligence with moral worth: animal…
Modified: February 25, 2022.
Modified: March 31, 2023.
Modified: .
It's a bit counterintuitive that high interest rates prevent inflation. After all, doesn't a high interest rate mean that the [ central bank…
Modified: October 09, 2022.
interesting things dennett says in "darwin's dangerous idea" the "baldwin effect" uses reinforcement learning to construct a version of…
Modified: February 10, 2022.
Interfaces enable modularity. In general, standardizing an interface can yield quadratic benefit at linear cost. Suppose we have people…
Modified: May 16, 2022.
Things to pay attention to in setting up a new place: Lighting makes a huge difference and can totally change the ambience of a space. Don…
Modified: January 19, 2022.
Fasting is a powerful, life-changing idea because it's simple, clear, easy to follow. A diet plan that involves counting calories requires…
Modified: July 11, 2021.
References: https://en.wikipedia.org/wiki/Intermolecular_force and linked pages conversations with GPT-4 A molecule is a set of atoms…
Modified: September 13, 2023.
The theory of musical intervals almost makes mathematical sense (but not quite): The sensible part: integer ratios The ancient Greeks…
Modified: July 09, 2022.
objectives: maximize entropy of the state visitation distribution requires empirical estimates of [ entropy ] maximize mutual information…
Modified: .
Modified: .
Modified: .
Modified: January 22, 2022.
Modified: August 08, 2021.
David Hume pointed out that there's no logical way to get from 'is' (descriptive) statements to 'ought' (normative) statements. This is…
Modified: .
The isoperimetric problem : among all closed curves in the plane with equal perimeter, which encloses the largest volume? It's well-known…
Modified: .
Update Feb 2025: I now think the existence of [ memory reconsolidation ] makes the point expressed here much less of a blackpill. Factual…
Modified: February 25, 2022.
Admitting things that I personally don't know is hard, because it feels like admitting a weakness or failing. But ignorance isn't a…
Modified: July 10, 2020.
"it's not my universe": a mantra for [ equanimity ] practice from [ Tucker Peck ]. If you believe in a God who does control the universe…
Modified: April 04, 2024.
Apparently Tucker Peck told this story in a talk. Sharon Salzberg says that if she could put one thing on her tombstone it would be this…
Modified: .
Notes copied from Google Docs https://docs.google.com/document/d/1G7Gxo-A3gQrlUx3G4BYmH-GjUWbB7eTOJZcuAs2ii0A/edit most of these things…
Modified: August 28, 2021.
Modified: .
From 2017: stay publicly active. work on open side projects, and/or publish, and/or blog. live at such a means that I could lose my job and…
Modified: February 15, 2022.
Something I've struggled with: given my position of immense privilege, how can I justify doing wasteful 'fun' things like skiing or clothes…
Modified: February 07, 2022.
I don't know if this makes sense, but one intuition I have for karma comes from the observation that the weights of a least-squares linear…
Modified: February 10, 2022.
We are given the opportunity to bet some fraction of our wealth on a coin flip with probability . We can repeat this as many times as we…
Modified: March 24, 2025.
multiple senses: in machine learning: positive definite (Mercer) kernels in linear algebra: kernel (nullspace) of a linear map in CS systems…
Modified: February 25, 2022.
Effects in the brain Ketamine is an antagonist of [ NMDA receptor ]s, and activates [ AMPA receptor ]s. References: Ketamine – Lorien…
Modified: February 13, 2022.
pg says: the random things that you learn as a kid make you into a key. Your job is then to find the lock that you fit into. But that's not…
Modified: June 12, 2021.
Adults tend to talk to and work with other adults. We don't spend nearly enough time addressing the problems of younger people: college…
Modified: April 01, 2022.
I just coined the phrase 'kitchen sink' deep learning for a vague idea that comes to me occasionally. Roughly: rather than using a uniform…
Modified: October 25, 2021.
An annoying thing about language-learning tutorials is that they often focus on language that you'll never actually use as a tourist. When I…
Modified: December 04, 2022.
Modified: .
Modified: .
Taco Cohen speculates on Large Control Policies as a successor to large language models: https://twitter.com/TacoCohen/status…
Modified: April 15, 2022.
Much of statistical practice is concerned with distinguishing signal from noise. For example, significance tests quantify the likelihood…
Modified: May 19, 2022.
If you believe that neural nets basically just memorize the training data, then training larger and larger models is hopeless. The…
Modified: September 06, 2021.
I don't want to be led. I want to be creative and do things that are dramatically new. Telling other people what to do feels almost evil to…
Modified: February 25, 2022.
Happens all the time, at small scales and large scales. Large scale: When I was young, it was possible to imagine becoming a confident…
Modified: March 08, 2020.
The set of good approaches is often hidden in an exponentially large space. Learning that an approach is not good doesn't help narrow that…
Modified: June 03, 2020.
Modified: September 11, 2021.
There's no easy or fast solution for feeling good about leaving academia, because you're giving up some aspect of your [ identity ], and…
Modified: February 07, 2022.
Modified: February 25, 2022.
(related: [ growing up means becoming wrong ], ask for evidence , [ Seeing Like A State ]) Related to some interactions with [ Josh…
Modified: February 25, 2022.
A thing that's tough about going through big personal changes is that it takes a while for your self-model to catch up with your actual self…
Modified: May 16, 2022.
With varying degrees of clarity and certainty. We are [ embedded agent ]s. So are any AI systems we build. We exist inside the world; the…
Modified: December 30, 2024.
Modified: .
People on the internet have very different standards as to when and how it's okay to ask a question. There are roughly two camps: Search…
Modified: March 19, 2022.
Modified: .
https://maskofreason.files.wordpress.com/2011/02/the-library-of-babel-by-jorge-luis-borges.pdf Insights illustrated by this story: Naming…
Modified: October 17, 2022.
Modified: March 07, 2020.
Modified: .
In principle we can apply [ automatic differentiation ] through any composition of differentiable operations. This lets us get gradients of…
Modified: July 21, 2022.
Modified: October 29, 2021.
I want to change the world. What does that mean? Suppose I create a billion-dollar company. That's an enormous amount of value. It's many…
Modified: February 14, 2021.
tags: created: 2023-12-07 modified: 2023-12-07 References: https://arxiv.org/abs/2006.16236 The usual [ transformer ] [ attention…
Modified: December 07, 2023.
A linear time-invariant system is one where the dependence of the output on the input is: linear: an input produces an output , and…
Modified: December 05, 2023.
Modified: .
podcasts and audiobooks Alan Watts 'out of your mind' lectures
Modified: February 15, 2022.