Note: All handouts and lecture slides are in PDF format. Any scheduling information posted for future dates should be treated as tentative.

Abbreviation: DHS - Duda et al.; PRML - Bishop's new book; HTF - Hastie et al.; NNPR - Bishop's older book; MK - MacKay.

• Wed 9/6/06
  • General introduction: goals of machine learning, examples, administrivia.
  • Lecture 1
• Fri 9/8/06 Note: change of location to CIT 368
  • loss, empirical/expected risk.
  • Linear regression; least squares.
  • Lecture 2
  • Regression demo shown in class: linRegSim1D.m (data to reproduce the figures)
  • A note summarizing some useful facts about matrices by Sam Roweis.
  • Recommended reading: PRML Sections 1.1, 3.1; HTF Chapter 3.

• Mon 9/11/06 Note: change of location to CIT 368

  • Optimal regression function
  • statistical regression models, likelihood and log-likelihood
  • Maximum likelihood estimation
  • Lecture 3
  • Notes (derivations etc.)
  • Recommended reading: DHS A.2 (appendix), HTF 3.1-3.3, PRML 3.1.

• Wed 9/13/06

  • Review of multivariate Gaussians
  • Uncertainty in ML estimate
  • Extensions of simple linear regression
  • Lecture 4
  • Notes (derivations etc.)
  • Matlab code to produce polynomial regression figures in the lecture: tryPolyFit.m, degexpand.m (and data in polyfitdata.mat)
  • Problem Set 1 out; you will also need meteodata.mat, apple_lda.mat, testWs.m.
  • Recommended reading: DHS A.4, A.5; PRML 3.1, Appendices B,C.

• Fri 9/15/06 In Lubrano, our new home

  • Introduction to classfication
  • A more detailed look at multivariate Gaussians
  • Linear Discriminant Analysis
  • Lecture 5
  • Another useful short summary by Sam Roweis, on Gaussian identities.
  • Recommended reading: NNPR 3.6; DHS A.4, A.5; HTF 4.1-4.3; PRML 2.3, 4.1, Appendices B,C.

• Mon 9/18/06

  • Fisher's LDA criterion
  • Decision theory; optimal classification
  • Lecture 6 (known typos fixed)
  • Notes (derivations etc.)
  • Recommended reading: PRML 1.5.1, 4.1, HTF 2.4, 4.1-4.3, NNPR 3.6, DHS 3.8.2.
  • Matlab code to play with Gaussian 1D marginals: margGausDemo.m, makeRandomSigma2d.m

• Wed 9/20/06

  • Decision theory: Bayes rule, optimal classification.
  • Generative models for classification
  • Discriminant functions
  • Lecture 7
  • Recommended reading: PRML 1.5.1, HTF 2.4, DHS 2.1-2.7.

• Fri 9/22/06

  • Estimation theory
  • Bias-variance tradeoff
  • Lecture 8

• Mon 9/25/06

  • Bias-variance tradeoff and model complexity
  • Naive Bayes classifier
  • Applications in document classification
  • Bayesian estimation, MAP
  • Lecture 9
  • Recommended reading: PRML 2.2.2, 3.2; HTF 7.2-7.3; DHS 9.3; NNPR 9.1.

• Wed 9/27/06

  • Bayesian estimation, MAP
  • Priors for Gaussian distribution
  • Discriminative models; logistic function
  • Problem Set 1 due.
  • Problem Set 2 Code: parseEmail.m, parseDirectory.m, logisticRegression.m
    Data: LogRegToy.mat, digitData.mat, dictionary.mat
  • Lecture 10
  • Recommended reading: PRML 4.3.2, 4.3.4, ...

• Fri 9/29/06

  • Logistic regression
  • Lecture 11
  • Recommended reading: NNPR 3.1.3, PRML 4.3.2-4.3.4.

• Mon 10/2/06

  • Logistic regression: gradient ascent algorithms.
  • Computational issues, convergence.
  • Overfitting and regularization.
  • Lecture 12
  • Recommended reading: TBD.

• Wed 10/4/06

  • Regularization.
  • Lecture 13

• Fri 10/6/06

  • Regularized regression: ridge regression and lasso.
  • Survey of what we have learned so far.
  • Large margin discriminative classifiers.
  • Lecture 14
  • Recommended reading: PRML 3.1.4, NNPR 9.2, HTF 3.4.

• Mon 10/9/06

  • No class - Columbus Day

• Wed 10/11/06

  • Support Vector Machines.
  • Problem Set 2 due.
  • Lecture 15
  • Recommended reading: PRML 7.1; HTF 12.2-12.3; DHS 5.11.
  • C. Burges, SVM tutorial.

• Fri 10/13/06

  • Guest lecture: optimization.

• Mon 10/16/06

  • No class
  • Problem Set 3 Code: genData.m, regularizationCode.m, plotLRcont.m
    Code from previous PSets: logisticRegression.m, degexpandScale.m
    Data: lrDataApricot.mat

• Wed 10/18/06

  • Guest lecture: unsupervised and reinforcement learning in robotics.

• Fri 10/20/06

  • Support Vector Machines: non-separable case, the kernel trick.
  • Lecture 16
  • Recommended reading: PRML 7.1; HTF 12.2-12.3; DHS 5.11.

• Mon 10/23/06

  • Non-parametric methods; nearest-neighbor methods.
  • Lecture 17

• Wed 10/25/06

  • Problem Set 3 due;
  • Midterm review.
  • Lecture 18

• Fri 10/27/06 CIT 367

  • Midterm (in class)
  • Problem Set 4
    Code: Code and data in a single .tar.gz

• Mon 10/30/06

  • Locally weighted regression.
  • Recommended reading: Atkeson et al., tutorial on LWR.
  • Mixture models; the EM algorithm.
  • Recommended reading: PRML Chapter 9; HTF 8.5; DHS 3.9; NNPR 2.6.
  • Lecture 19

• Wed 11/1/06

  • The EM algorithm for Gaussian mixtures.
  • Recommended reading: PRML Chapter 9; HTF 8.5; DHS 3.9; NNPR 2.6.
  • Lecture 20

• Fri 11/3/06

  • General view of EM; model selection.
  • Recommended reading: PRML Chapter 9; HTF 8.5; DHS 3.9; NNPR 2.6.
  • Lecture 21

• Mon 11/6/06

  • Model selection, minimum description length.
  • Recommended reading: Hansen and Yu
  • Lecture 22

• Wed 11/8/06

  • Unsupervised learning, clustering, K-means.
  • Recommended reading: PRML 9.1, HTF 14.3
  • Lecture 23

• Fri 11/10/06

  • Clustering: hierarchical, spectral.
  • Recommended reading: HTF 14.3, DHS 10.9, 10.12
  • Lecture 24
  • Problem Set 4 due

  • Problem Set 5

    
    Code: Code and data in a single .tar.gz

• Mon 11/13/06

  • Spectral clustering.
  • Lecture 25

• Wed 11/15/06

  • Dimensionality reduction; Principal Component Analysis
  • Recommended reading: HTF 14.5, NNPR 8.6, PRML 12.1
  • Lecture 26

• Fri 11/17/06

  • Feature selection.
  • Lecture 27

• Mon 11/20/06

  • Stepwise regression; boosting.
  • Lecture 28

• Wed 11/22/06

  • AdaBoost
  • Recommended reading: PRML 14.3, HTF 10.1-10.6
  • Lecture 29
  • Problem Set 5 due
  • Project proposals due (200-level)

• Fri 11/24/06

  • No class - Thanksgiving recess

• Mon 11/27/06

  • Mixtures of experts.
  • Markov Models.
  • Recommended reading: PRML 14.5, 13, HTF 9.5
  • Recommended reading: Rabiner's tutorial on HMMs for speech recognition.
  • Recommended reading: Shannon's paper on prediction and entropy of English, 1951.
  • Lecture 30

• Wed 11/29/06

  • Hidden Markov models; forward-backward algorithm.
  • Recommended reading: PRML 13.1-13.2, DHS 3.10
  • Lecture 31

• Fri 12/1/06

  • Hidden Markov models: estimation, Baum-Welch algorithm.
  • Recommended reading: PRML 13.1-13.2, DHS 3.10
  • Lecture 32

  • Problem Set 6

    
    Code: Code and data in a single .tar.gz

• Mon 12/4/06

  • Hidden Markov models: decoding (Viterbi).
  • Lecture 33
  • Graphical models.

• Wed 12/6/06

  • Graphical models, inference.
  • Lecture 34

  • Problem Set 7

    
    (not graded; do not hand it in)

• Fri 12/8/06

  • Advanced topics (beyond cs195-5).
  • Lecture 35
  • Reading Period begins

• Mon 12/11/06

  • Advanced topics
  • Lecture 36

• Wed 12/13/06

  • Pre-final review
  • Lecture 37
  • Problem Set 6 due

• Fri 12/15/06

  • No class.

• Mon 12/18/06

  • Final at 9:00am, Wilson 101