Modified: February 17, 2022
substantive questions I've had
This page is from my personal notes, and has not been specifically reviewed for public consumption. It might be incomplete, wrong, outdated, or stupid. Caveat lector.substantive questions I've had
these are things I've wondered about that were never answered properly in the classes in which I learned them.
statistics / probability / learning
why is least-squares special? why not minimize some other error function?
- one answer: minimizing least squares is equiv. to maximizing likelihood given Gaussian error
why is the logistic function special?
- one answer: falls out naturally as the response function when you create a GLM with Bernoulli random variables. this is similarly where the softmax function comes from (for a multinomial GLM)
why are Gaussians special?
- empirically useful, but this can be justified by:
- central limit theorem
- limit of binomials
what are the limits of neural networks? what sort of things can they learn to do?
- answers are subtle and I don't really understand them
why do kernels need to be positive definite?
- because this allows kernel matrices to be factored K = PP^T (using e.g. the SVD or Cholesky decompositions), thus representing the kernel values as dot products of the rows of P.
what is the pseudo likelihood?
- it’s the likelihood you get from leave-one-out cross validation, i.e. replacing the likelihood of the data set as a whole, with the product of likelihoods (or sum of log-likelihoods) of each data point given all the others.
algorithms
Q: when we analyze algorithms, what counts as a "step"? A: we declare a computational model (sometimes implicitly). This is usually something like a RAM machine (which is polynomially faster than a Turing machine).
calculus Q: what exactly can I do with infinitesimals, e.g. dy/dx. When are they like fractions (can cancel, etc.) and when are they not?
A: I don't really know, but apparently there are whole calculus books which develop calculus rigorously in terms of infinitesimals. For example, http://www.math.wisc.edu/~keisler/calc.html ("Elementary Calculus: an Infinitesimal Approach"). I think this is the same as nonstandard analysis, and the infinitesimals are the same as hyperreals.
Apparently there's more clarification in Spivak's Calculus on Manifolds.