Least Square Problem

• 用观测值和真实值的平方差作为优化目标的问题。

$$ \min {\beta} \sum{i=1}^{n}\left(y_{i}-x_{i}^{\top} \beta\right)^{2} $$

Least absolute deviation

• 用观测值和真实值差的绝对值为优化目标

$$ \min {\beta} \sum{i=1}^{n}\left|y_{i}-x_{i}^{\top} \beta\right| $$

Regularized least squares

• 加入一些正则项
• Ridge least squares：使用参数的二范数 - $\min {\beta} \sum{i=1}^{n}\left(y_{i}-x_{i}^{\top} \beta\right)^{2}+\lambda\|\beta\|_{2}^{2}$
• Lasso least squares：使用参数的一范数 - $\min {\beta} \sum{i=1}^{n}\left(y_{i}-x_{i}^{\top} \beta\right)^{2}+\lambda\|\beta\|_{1}$

范数

• 0范数：稀疏度

Desire solution β to be sparse aka with small ∥β∥0 i.e. few non-zero coefficients.

Why? Only few features are relevant, require correspondingly few data points, . . .