Scope and goals of the class

This course introduces the fundamental principles of statistical learning and their implications for the basic problems of modeling and inference. The emphasis is on algorithms and models useful in a broad range of applications, and on understanding the underlying assumptions, weaknesses and strengths of each approach. The ultimate goal is for the students to become familiar with arsenal of modern machine learning, so that as practitioners they could judge when a particular tool is appropriate, and how it could be adapted to the problem at hand. Developing such practical intuition is the main desired outcome of taking the course.

In light of this practical outlook, discussion of efficient computation methods will receive significant attention in the course. Part of the homework assignments require implementing learning algorithms in Matlab and evaluating them on data sets, many of which will be taken from real-life problems. The emphasis in such assignments is not on massive software engineering but rather on algorithmic clarity and efficiency of the implementation.

Rough list of topics covered

• Basic concepts in statistical learning: loss, risk, likelihood.
• Empirical risk minimization and generalization.
• Supervised learning:
  • Parametric regression.
  • Additive models; logistic regression.
  • Generative models; naive Bayes classifiers.
  • Discriminative methods:
    • The perceptron and neural networks
    • Linear discriminative analysis
    • Support vector machines
  • Mixtures of experts.
• Unsupervised learning:
  • Clustering: agglomerative, K-means, spectral.
  • Mixture models and the EM algorithm.
  • Dimensionality reduction.
• Mutual information and entropy and their connection to learning.
• Feature selection.
• Complexity and model selection.
• Ensemble methods; boosting.
• Basics of inference and learning in graphical models.
• Hidden Markov models and Markov random fields.

Helpful books

None of these books is absolutely necessary, and the lectures and problem sets will fully contain the material required to succeed in the course. Moreover, none of these books contains 100% of the material covered in cs195-5. However, reading certain parts of each of these books may be useful to understand the material; we will indicate some of these suggested readings in the course calendar.

	C. M. Bishop Neural Networks and Pattern Recognition Still a great reference, but may be superceded by the new book on the right		C. M. Bishop Pattern Recognition and Machine Learning
	R. O. Duda, P. E. Hart and D. G. Stork Pattern Classification		D. McKay Information Theory, Inference, and Learning Algorithms Available online
	T. Hastie, R. Tibshirani and J. H. Friedman The Elements of Statistical Learning