Modified: October 29, 2021.
I want to change the world. What does that mean? Suppose I create a billion-dollar company. That's an enormous amount of value. It's many…
Modified: February 14, 2021.
tags: created: 2023-12-07 modified: 2023-12-07 References: https://arxiv.org/abs/2006.16236 The usual [ transformer ] [ attention…
Modified: December 07, 2023.
A linear time-invariant system is one where the dependence of the output on the input is: linear: an input produces an output , and…
Modified: December 05, 2023.
Modified: .
podcasts and audiobooks Alan Watts 'out of your mind' lectures
Modified: February 15, 2022.
I've heard it said, and it's been ringing true to me, that the thing to do is live the life you want to have now , not plan to spend years…
Modified: February 12, 2022.
Some things are genuinely hard to do. But many others I don't do just out of laziness, or maybe lack of [ agency ]. I know that they're…
Modified: September 12, 2021.
Sources: https://www.lesswrong.com/posts/y5GftLezdozEHdXkL/an-intuitive-guide-to-garrabrant-induction https://www.lesswrong.com/posts/jtMXj…
Modified: April 02, 2025.
A atomic orbital with two electrons both attached to the same atom. In contrast to a bond, where each atom contributes one electron. Lone…
Modified: May 14, 2021.
Notes on https://www.pragmatic.ml/a-survey-of-methods-for-incorporating-long-term-context/ 'Standard' transformers have O(n**2) complexity…
Modified: March 21, 2020.
Modified: .
There's a tendency to focus on things that we have the (conceptual/mathematical/societal) tools to understand, even when we know this is…
Modified: February 10, 2022.
How do I justify being only 'mostly' [ vegetarian ]? I know that cows and chickens are abused to produce milk and eggs. Why is avoiding…
Modified: February 20, 2021.
Modified: .
What does it mean to [ love ] someone? Of course this question has as many answers as there are people, and probably more. But here's one…
Modified: November 28, 2023.
What does it mean to love someone? Of course this question has as many answers as there are people, and probably more. But here's one view…
Modified: November 28, 2023.
Modified: November 28, 2020.
see also [ metta ]
Modified: April 10, 2024.
I saw this phrase on Twitter somewhere and it really resonates as a description of the ideal approach to science. There is no real…
Modified: May 16, 2022.
A macrostate in statistical mechanics is a collection of base-level states; equivalently, a subset of [ phase space ]. It's what you see…
Modified: April 13, 2022.
Modified: .
Mahamudra means the 'great seal' or 'great gesture'. We take and [ hold the view ] that each event arising in [ awareness ] --- every sight…
Modified: October 09, 2022.
Managing for high-variance / creative work versus low-variance consistent work: https://blog.sbensu.com/posts/2023-01-18-high-variance…
Modified: January 23, 2023.
There are a lot of difficult decisions to be made in life. Maybe you need to decide the business strategy of a company, knowing that good…
Modified: November 02, 2022.
An idea I got from [ John Higgs ]'s discussion of metamodernism is that taking [ all models are wrong ] to its logical conclusion requires…
Modified: January 06, 2023.
Sometimes I've been scared of losing my identity. In particular I worry about working a non-research job, or having sex with (or being…
Modified: February 07, 2022.
I have a private theory about what marijuana does. I'll try to articulate it here. I don't know much about the public theories, so maybe…
Modified: January 17, 2021.
A martingale is any [ stochastic process ] that stays the same in expectation. Formally, is a martingale if This condition is related to…
Modified: August 27, 2022.
How to think about giving a good massage? Know which way the muscle fibers go. For deep release, exert force perpendicular to the muscle…
Modified: September 02, 2024.
Modified: .
Reviewing this 3blue1brown video: https://www.youtube.com/watch?v=O85OWBJ2ayo The matrix exponential is written as E to the power of a…
Modified: November 20, 2023.
The Woodbury-Morrison-Sherman matrix inversion lemma, is sometimes useful just for algebraic simplifications. In cases where and are…
Modified: March 16, 2022.
Notation for Matrix Multiplication Let and . Then just by the definition of matrix multiplication (the summation over is performing…
Modified: March 16, 2022.
References: Hu, Yang (2022) Feature Learning in Infinite-Width Neural Networks https://arxiv.org/abs/2011.14522 Yang, Hu et al. (202…
Modified: December 29, 2023.
For any reward function and policy , consider the entropy-regularized reward Taking as our objective the (expected, discounted…
Modified: July 28, 2022.
Note: these are personal notes, taken as I was refreshing myself on this material. They're mostly stream of consciousness and probably not…
Modified: March 16, 2022.
A function is measurable with respect to [ sigma-algebra ]s on its domain and on its range if the pre-image of any event is…
Modified: August 27, 2022.
Modified: .
The core insight that got me interested: "moments of recognizing your thoughts drifting and bringing them back to your breath" are not…
Modified: October 03, 2021.
Apr 24, 2023 be clear on what you're doing in each session. don't mix concentration and emptiness I guess this is related to Tucker's…
Modified: December 13, 2023.
Dec 13, 2023 Sangha session with Dustin: Something I realized during a concentration practice is that being honest about how ‘well’ the…
Modified: December 13, 2023.
Generally I think the dharma is deeply true and that [ meditation ] done right is healthy and potentially very beneficial. But I struggle…
Modified: July 19, 2024.
Specific states or abilities that can arise from skillful meditation: feeling [ equanimity ] first cessation seeing nimitta accessing…
Modified: .
Modified: .
Modified: .
To train a [ transformer ] layer on a sequence of length requires the output of the attention computation where are matrices and is…
Modified: February 19, 2024.
Suppose we want to do [ automatic differentiation ] on a [ computational graph ] of sequential length . This could equally well be a…
Modified: January 02, 2024.
Described, among other places, in Unlocking the Emotional Brain . Insofar as much of Buddhism is about dissolving [ samskara ]s…
Modified: February 04, 2025.
One last thought mental models are so, so important. When I think about computer modeling. It's actually great computers are powerful they…
Modified: July 25, 2020.
Like democracy , meritocracy is the worst form of social organization, except for all the others that have been tried. Of course it is good…
Modified: December 01, 2022.
References: Risks from Learned Optimization in Advanced Machine Learning Systems A [ reinforcement learning ] algorithm attempts to find the…
Modified: March 28, 2023.
Chemically, a substituted [ phenethylamine ]. Like [ dopamine ] but with methyl groups hanging from the two oxygens, and another oxygen…
Modified: December 21, 2022.
Generally this means training some aspect of the learning procedure itself. There is then an inner-loop learning procedure, which follows…
Modified: October 04, 2021.
Unlike most modern [ deep learning ] systems, humans: don't have separate training/test phases (though we may have wake/[ sleep ]) don't…
Modified: January 16, 2022.
Stuart Russell told the story of giving a talk on meta-reasoning at Stanford, with Don Knuth in the audience, where he opened with a slide…
Modified: October 17, 2022.
n-methyl-[ amphetamine ]
Modified: July 24, 2023.
Modified: .
Pali (Buddhist) term for [ loving-kindness ]
Modified: April 10, 2024.
From @visakanv on Twitter: (relevant to [ nothing matters ])
Modified: August 13, 2022.
Modified: .
A lesson from [ Tucker Peck ]: unresolved questions are the worst thing in meditation. For example, you're just sitting down to practice…
Modified: November 27, 2023.
Considering a bilevel optimization problem (or saddle point problem) on the two-argument function , in general it holds that That is, the…
Modified: July 07, 2022.
Short descriptions of things, when they exist, must capture some kind of structure. The principle of [ Occam's razor ] posits that we should…
Modified: April 12, 2022.
Mirror descent is a framework for optimization algorithms: many algorithms can be framed as mirror descent, and proofs about mirror descent…
Modified: October 03, 2020.
What pieces of [ mirror descent ] can we automate? See also [ natural gradient implementations ] Given a mirror function , we can compute…
Modified: September 07, 2020.
Modified: June 12, 2021.
(originally from 2020-04-29) On another note, last night I tried to dictate (on Otter) my sense of my life goals. I came up with a very…
Modified: January 24, 2022.
Modified: July 31, 2021.
[ Otter notes ]: Can I explain what a mixed effects model is from a graphical model standpoint? On the inference side, I think it's just…
Modified: January 23, 2022.
A mixture-of-experts model consists of a set of functions , the 'experts', and a gating function that determines how to select which…
Modified: .
I have a [ strong opinion weakly held ] that doesn't seem to be wildly shared in the [ approximate Bayesian inference ] community: reverse…
Modified: March 14, 2022.
Original paper: Finn, Abbeel, and Levine, ICML 2017, https://arxiv.org/abs/1703.03400 An approach for [ meta learning ] that works with any…
Modified: February 20, 2022.
Modified: .
Often we don't explicitly use 'model-based RL' methods, instead people in robotics talk about Sim2Real: adapting a policy pretrained in a…
Modified: July 20, 2022.
Modified: .
Stack: goal: sample from conformations of arbitrary hydrocarbons (or whatever). simpler goal: sample from conformations of ethane. simpler…
Modified: May 16, 2020.
Naively you might think that the government just decides how many dollars there should be, and that's that. This is not true. Since [ IOUs…
Modified: February 09, 2022.
A monoamine oxidase (MAO) is an enzyme that breaks down mono-[ amine ] neurotransmitters such as [ dopamine ], [ serotonin…
Modified: May 22, 2022.
A very natural form of [ meta-reasoning ] that selects the most promising computations. The simplest form of 'expanding' a node assumes a…
Modified: March 22, 2022.
There is a connection between moral realism and belief in [ qualia ]. If you see "experience" ([ awareness ]) as a real, fundamental aspect…
Modified: .
I don't hold the moral view that it's better to be a morning person than an evening person. Having always tended towards a later sleep…
Modified: March 14, 2023.
In any human-to-human interaction, language carries some very important high-order bits, but it can only carry a few bits. It can help…
Modified: June 12, 2021.
This is one of the big problems with the world. Not the only one, and not the only way to look at it. But it's everywhere. status : a…
Modified: January 25, 2022.
Modified: .
(see David Graeber https://www.strike.coop/bullshit-jobs/ ) Most work is oriented towards achieving [ instrumental goal ]s. But most…
Modified: February 25, 2022.
I like this take on working with procrastination from a [ nondual ] [ awareness ] perspective: From the viewpoint of "the beyond…
Modified: June 21, 2024.
maneuvering: the bike goes where I look. look around the turn I want to do. keep elbows up. shift body weight to counterbalance the bike. E…
Modified: May 12, 2022.
possible refs: google's multimodal architectures: https://webcache.googleusercontent.com/search?q=cache:https://towardsdatascience.com…
Modified: September 25, 2023.
From a conversation I had about [ attention ] mechanisms in deep architectures. Maybe that terminology is too suggestive --- it's just a…
Modified: March 03, 2024.
We say that a random vector is multivariate Gaussian with mean and covariance matrix if it can be written where is a vector if i.i.d…
Modified: March 16, 2022.
[ thoughts on multivariate causalimpact ]
Modified: February 15, 2022.
This was originally a section of breakup.org, written several years ago. this is more related to jobs and identity, but for cases when I get…
Modified: May 22, 2021.
I want to intentionally spend my time well. I remember back in grad school I would spend evenings reading papers, just as a form of growth…
Modified: February 25, 2022.
I've identified as a 'tech' person, but I now feel uncomfortable in many tech circles. What is tech and what does it mean to be a tech…
Modified: January 24, 2022.
It's a useful exercise to occasionally reflect on what I value. stab 1: Generally pro tech, creating new things, non-zero-sum contributions…
Modified: November 27, 2023.
Modified: .
Modified: February 09, 2020.
Recommended by Michael Edward Johnson:
Modified: September 28, 2023.
A 'natural' abstraction is one that we expect any agent (or at least, a wide range of agents) to develop because it gets at something…
Modified: May 04, 2023.
Modified: .
We don't typically think of it this way, but you can derive a [ gradient descent ] step as finding the point that minimizes a linearized…
Modified: July 06, 2022.
How can we automate [ natural gradient ]? See also [ mirror descent implementations ]
Modified: September 19, 2020.
Cool trick: some applications can improve on nearest-neighbor lookup by training 'Exemplar SVM's. Instead of matching against a set of…
Modified: April 15, 2023.
Modified: .
My position (a [ strong opinion weakly held ]) is that global utility is currently negative, and probably always has been. It's conceivable…
Modified: August 25, 2022.
A negligible function is a function such that, for any positive integer there exists an integer such that for all , i.e., that…
Modified: October 23, 2022.
Christian Naesseth, Fredrik Lindsten, Thomas Schon (2015): http://proceedings.mlr.press/v37/naesseth15.html The main idea: In an SMC…
Modified: July 14, 2021.
Like the proverbial half-full glass, smart people can look at the same reality of the current capacities of neural nets, and come to…
Modified: April 07, 2020.
Sometimes you'll see people say that neural nets 'just' memorize and interpolate their training data. No one denies that neural nets with…
Modified: .
Parts of a neuron: dendrites: these branch out to receive connections from other cells axons: these branch out to send signals to other…
Modified: August 08, 2021.
Modified: .
Modified: February 25, 2022.
The folklore no-free-lunch 'theorem' in machine learning says that, for any pair of learning algorithms, there exists some dataset on which…
Modified: March 04, 2022.
Modified: .
No-self is one of the [ three characteristics ] that traditional Buddhism holds are present in all phenomena. In later Buddhism, the…
Modified: May 30, 2023.
https://arxiv.org/abs/1712.02390 Basic idea: optimizers like Adam and RMSProp already keep track of posterior curvature estimates. These are…
Modified: October 30, 2020.
Instead of directly targeting a specific rate of inflation, a [ central bank ] may target a fixed rate of nominal GDP growth, which is equal…
Modified: .
One way to model real-world [ causality ] is a bunch of forces working with and against each other. In this view, no individual force…
Modified: July 14, 2023.
NFTs 101: https://medium.com/@intenex/nfts-101-why-nfts-are-a-generational-innovation-4626ae803e3b Among many other things, NFTs are…
Modified: October 05, 2021.
Modified: .
Modified: .
Obligatory disclaimer: there will never be a drug to turn you into Einstein. Most of effective high-level thinking lies in 'software…
Modified: July 09, 2022.
Modified: August 08, 2021.
References: Gu et al., Continuous Deep Q-Learning with Model-based Acceleration (2016). Instead of modeling directly, we build a network…
Modified: July 19, 2022.
Something can be true but not 'true enough'. That is, you have a compelling causal theory for why X should increase Y. It might be that the…
Modified: August 21, 2020.
I've started reading The Art of Doing Science and Engineering by Richard Hamming. History of computing: Analog computing goes back forever…
Modified: June 03, 2020.
Because: [ goals are arbitrary ]: achieving a goal, or failing to, doesn't really matter because the goal was arbitrary anyway. From the…
Modified: February 25, 2022.
There's a spiritual idea, in Buddhism and elsewhere, that there is "nothing to do": everything is already suffused with "primordial…
Modified: June 06, 2023.
Modified: August 02, 2020.
Modified: February 10, 2022.
Modified: .
Don't invert that matrix: https://www.johndcook.com/blog/2010/01/19/dont-invert-that-matrix/ Seven sins of numerical linear algebra…
Modified: December 28, 2022.
A very incomplete and maybe nonsensical intuition I want to explore. Classically, people talk about very simple [ reward ] functions like…
Modified: March 31, 2023.
A few (relatively uninformed) thoughts about on- vs off-policy [ reinforcement learning ]. Advantages of on-policy learning: On-policy…
Modified: April 23, 2022.
Original: Daily reflections What am I grateful for today?:: Some goals : Goals for the next ~year:: Goals for the next ~month:: Goals for…
Modified: January 23, 2022.
Modified: .
The brain doesn't have separate models of each of the [ sense gate ]s (and thought). Instead it just stores each moment of perception as a…
Modified: .
Informally, a function is a one-way function if it is easy to compute but hard to invert. Or more generally, hard to pseudo-invert, i.e…
Modified: October 23, 2022.
These are things that I might plausibly decide I want to work on when I sit down on the weekend. Expanding nodes on this graph. Blogging…
Modified: February 22, 2020.
How do we maintain values when our models of the world shift? If someone's goal in life is to "do God's will", and then they come to believe…
Modified: April 12, 2023.
As Josh Marshall said , at the beginning of the Trump presidency: "Optimism is not primarily a prediction but an ethic, a philosophy, a way…
Modified: June 08, 2021.
Modified: October 26, 2021.
If is a [ martingale ] and is a [ stopping time ], then any of the following conditions implies that : The stopping time is bounded…
Modified: August 29, 2022.
Modified: May 01, 2020.
Ken McLeod claims that 'emotional reactivity' is the origin of suffering. Pain consists both in what happens and in our reaction to it. But…
Modified: October 06, 2021.
Modified: March 02, 2022.
Modified: .
mnemonic: OIL RIG = 'oxidation is losing (electrons), reduction is gaining (electrons)' in contrast to [ acid-base chemistry ], which is…
Modified: July 31, 2021.
This is how [ mitochondria ] produce most of their [ ATP ]. Mitochondria have an outer membrane and an inner membrane, so there are two…
Modified: July 31, 2021.
Modified: .
Look again at that dot. That's here. That's home. That's us. On it everyone you love, everyone you know, everyone you ever heard of, every…
Modified: November 30, 2022.
Modified: .
Modified: July 10, 2020.
References for PDEs: commutant's Youtube videos: https://www.youtube.com/playlist?list=PLF6061160B55B0203 Fundamental PDEs wave equation…
Modified: June 07, 2024.
Basic notes from https://www.stats.ox.ac.uk/~doucet/andrieu_doucet_holenstein_PMCMC.pdf Setup: we have parameters and time series model…
Modified: April 06, 2020.
Chocolate tasting: buy a bunch of high-end, single-origin chocolate bars. Parcel them out blind. Give people a pad to take notes on what…
Modified: May 19, 2020.
We often see optimization problems with objectives of the form where is the main function of interest (e.g., training loss in machine…
Modified: July 15, 2022.
“Remember that a person’s name is to that person the sweetest and most important sound in any language.” Dale Carnegie (How to Win Friends…
Modified: March 02, 2022.
When you're thinking about doing something that feels right to you, it's easy to get caught up in worrying about what other people will…
Modified: February 10, 2022.
reading the perceiver papers from Deepmind: Perceiver: Jaegle et al 2021 https://arxiv.org/abs/2103.03206 Perceiver-IO: Jaegle et al 202…
Modified: September 25, 2023.
In the [ 5-MeO-DMT ] trip where I experienced [ ego death ], I saw a [ magical display ] of beautiful colors and flowing motion and…
Modified: February 10, 2022.
The AI Effect refers to the widely-recognized phenomenon that 'once we know how to do it, it's not AI'. For example, playing chess well…
Modified: May 29, 2020.
I always found it weird that philosophy spends so much time talking about specific historical philosophers. Who cares what Aristotle, or…
Modified: January 24, 2022.
When considering one's impact on the world, it's important (? or at least tempting) to think about about your value-over-replacement. If you…
Modified: July 07, 2023.
(see also: [ large models ]) There's a viewpoint that neural nets just memorize the training data, so the more training data you have, the…
Modified: February 10, 2022.
Modified: .
Modified: .
Modified: May 14, 2021.
Developed and widely used in Russia, phenibut is an analogue of [ GABA ] with a phenyl ring substituted at the carbon, giving it the name…
Modified: September 28, 2023.
Why Nature Chose Phosphates (science.org)
Modified: January 19, 2022.
To His Coy Mistress Andrew Marvell, 1681 Had we but world enough and time, This coyness, lady, were no crime. We would sit down, and think…
Modified: July 19, 2024.
The paradoxical thing about pointing-out style meditation teaching is that you can't really explain the instructions when they're unclear…
Modified: October 30, 2021.
Modified: .
Modified: March 02, 2022.
(see also my [ deep RL notes ] from John Schulman's class several years ago, which cover much of the same material) We can approach…
Modified: March 14, 2024.
Modified: .
There are a few ways to do this. Google's PaLM uses rotary embeddings so it seems like that's probably close to the state of the art? But…
Modified: September 28, 2023.
Modified: .
Different experimental conditions may give rise to different outcomes . For example, let the variable indicate whether a person is…
Modified: August 06, 2021.
Prayer is a form of [ therapy ]. It's about clarifying your values: figuring out what you really want so that you can ask God for it. and…
Modified: February 07, 2022.
A [ stochastic process ] is predictable if its value at time is fully determined by information available at time . Any fully…
Modified: August 27, 2022.
A really valuable exercise that I should consider building into my routine is to regularly try to make and write down explicit predictions…
Modified: January 24, 2021.
Consider an agent that is purely concerned with [ predictive processing ]: finding the optimal [ compression ], or equivalently the optimal…
Modified: April 12, 2023.
The theory of predictive processing seems to be attracting a lot of interest in neuroscience and [ meditation ] circles. I want to try to…
Modified: .
https://www.quora.com/What-is-a-preference-cascade A lot of how people act is driven by how they think they're 'supposed' to act. There's…
Modified: February 22, 2022.
AI / RL Distributional RL book: https://www.distributional-rl.org/ Alignment Sequences: Value learning: https://www.alignmentforum.org/s…
Modified: June 27, 2023.
Modified: .
A Bayesian view of (one aspect of) [ attention ] inspired by a conversation with Shamil Chandaria on [ predictive processing ]. (but this…
Modified: May 25, 2023.
It seems like there is, or can be, a virtuous relationship between privacy and generalization. You don't want to memorize too many…
Modified: February 14, 2021.
Illegible privilege We often talk about the 'privilege' associated with certain categories: being born white, straight, male, rich, in a…
Modified: April 04, 2024.
(I got this concept from SuccessfulFriend.) As people grow up and form their identities, they need models, and not just models; they need…
Modified: February 22, 2022.
Can we think about [ generative flow network ]s as a potentially tractable formulation of probabilistic program induction?! executing a line…
Modified: March 14, 2022.
Modified: February 08, 2020.
Many [ probabilistic programming ] researchers frame their work as part of the broader problem of [ artificial intelligence ]. Artificial…
Modified: December 01, 2023.
A short note on interpreting a transformer layer as performing maximum-likelihood inference in a Gaussian mixture model: https://arxiv.org…
Modified: October 30, 2020.
Matt Levine explains how a financier might react to losing a billion dollars: Sure sure the risks didn’t work out but you probably have a…
Modified: February 22, 2022.
A probability space consists of: A set of outcomes aka possible worlds; these represent all the ways the world might be. This is the…
Modified: August 27, 2022.
(aka, why frequentists will always make more money) In the "real" (corporate/governmental) world, most high-level decision making is…
Modified: March 04, 2022.
Modified: September 27, 2024.