generalization: Nonlinear Function
Created: February 14, 2021
Modified: October 02, 2021

generalization

This page is from my personal notes, and has not been specifically reviewed for public consumption. It might be incomplete, wrong, outdated, or stupid. Caveat lector.
  • Fundamentally, where does generalization come from?
    • causality: a model may generalize because it has discovered the true mechanism, or an abstract mechanism that captures some of the true mechanism.
    • But not all generalization comes from causality. Linear regression can generalize even when we know that the matrix multiplication is not the causal process underlying the data. Sure, the true causal process viewed from some angle might produces results that are roughly linear in some input, so we can think of linearity as an 'abstract mechanism', but that doesn't feel like a full or satisfying story.
    • Generalization can come from smoothness. Things in physics move smoothly, so almost anything produced by a physically realistic causal mechanism will be mostly smooth. Linearity, or (more generally) polynomials of limited degree, is a form of smoothness assumption.
  • The no free lunch theorem: without assumptions, generalization is impossible.
Links to this note

AI reflections master

This page is a general jumping-off point for organizing my thoughts about the [ AI research landscape ], where the field is, where it is…

state values, then action values

A common pattern in [ reinforcement learning ] pedagogy is to develop some idea first in the context of estimating state values , and then…

free lunch theorem

'[ no free lunch theorem ]' arguments are misleading because they consider the space of all possible functions. In fact, we usually care…

Occam generalization bound

Here's a formal argument for [ Occam's razor ], adapted from Theorem 2.2 of Kearns and Vazirani's "An Introduction to Computational Learning…

distinction

[ Otter notes ] August 2020: When somebody says that X is good---here X could be love relationships, money, peace, or whatever---it is never…

data efficiency

Current (2021) deep networks require huge datasets in order to [ generalization|generalize ]. But we know that humans can do one-shot…

trapped priors

SSC link: How general is this phenomenon? You have a belief Your belief colors your perception of something that doesn't inherently…

double descent

Empirically, as model capacity increases past the memorization threshold ( ), [ generalization|generalization ] error starts decreasing…

privacy

It seems like there is, or can be, a virtuous relationship between privacy and generalization. You don't want to memorize too many…

teaching machine learning

Rob wants to firm up his foundations. He wants to understand relevant stats, probabilistic models, inference, and maybe work our way up to…

Meta
Tagged With: