Course Requirements and Grading
Lab(30%)
Python
Synthetic data
2 deliverables, distributed over moodle
Theory exercises(0/20)
- close to the end(early December)
Final exam(70%)
Theory questions(judgement-oriented)
Simulate running algorithms by hand
Meeting hours
Office:104B, 68-72 Gower street
Meeting hours: Tuesday, 14:00-15:00
Prerequisites:
Linear Algebra; Calculus; Probability; Programming
Machine Learning
data -> maodel ->prediction
Least squares model
least squares solution for linear regression
: probleim dimension, e.g. 1D, 2D( can visualize)
: training set size
Training set: input-output pairs where,
, generally can be
: weight,
: noise
other notation:
Remark: ";" represent column vector
linear regression model
that is
Loss function:
goal:
Least squares solution for linear regression:
Generalized linear regression model
where
can be other form besides
( if
, and
, it is just the linear regression model )
If , then it is k-th degree ploynomial fitting
If the highest order of is 2, then it is second-order polynomials fitting
set where
can be other form besides
( if
, and
, it is just the linear regression model )
If , then it is k-th degree ploynomial fitting
If the highest order of is 2, then it is second-order polynomials fitting
set
then the model is:
Least squares solution for generalized linear regression:
approximations
If (e.g. 30 points, 2 dimensions): overdetermined system
If (e.g. 30 points, 3000 dimensions): underdetermined system (overfitting)
How to control complexity( Regularized linear regression)
1.use vector norm (L2, L1, Lp norm) to measure residual vector
Remark: different norm represent different regularized linear regression, here we use L2 norm
2.rewrite loss function:
this is ridge regression, a.k.a, L2-regularized linear regression
Remark: is "hyperparameter", select
with cross-validation(use cross-validation for diff values of
-- pick value minimizes cross-validation error)
Cross-validation: least glorious, most effective of all methods (teacher said)
3.Least squares solution for ridge regression:
