CPSC 440 and 540 - Advanced Machine Learning (January-April, 2022)
Lectures: Mondays, Wednesdays, and Fridays (3-4, beginning January 10)
Lectures will initially be held on Zoom. The link will be made available on Canvas (for registered students), and will be been e-mailed to students on the wait list.
Beginning February 7th, we will move to in-person instruction at Woodward Instructional Resources Centre Room 6.
Tutorials: Mondays and Wednesdays (4-5, beginning January 17)
Links will be made available as above, and we may eventually move to DMP 101.
Instructor: Mark Schmidt.
Office hours: Fridays (beginning at 4, in ICICS 146)
Synopsis: This course is intended as a second or third university-level course on machine learning,
a field that focuses on using automated data analysis for tasks like pattern recognition and prediction.
The class is intended as a continuation of CPSC 340 (or 532M), and will assume a strong background in math
and computer science. Topics will (roughly) include deep learning, generative models, latent-variable models,
Markov models, probabilistic graphical models, and Bayesian methods.
Registration: Graduate and undergraduate students from any department are welcome to take the class. Undergraduate students should enroll in CPSC 440
while graduate students should enroll in CPSC 540. Below are more details on registration for each course:
- The majority of the seats in 440 are reserved for UBC computer science majors. For other students,
to enroll in the course you need to sign up for the wait list.
- The seats in 540 are reserved for graduate students in CPSC, EECE, or STAT. If these seats are full or you are from a different department, to enroll in the course you need to sign up for the wait list.
Please read about the differences between 340/532M and 440/540 below,
and do not take this course "because 340/532M is full" or not offered in a given term.
Note that in previous courses I have taught that all students on the wait list were ultimately accepted into the course (but we have not always had room for auditors).
You will need to be on the waiting list by January 17th to register.
Starting in the second week of classes, we'll have weekly tutorials run by the TAs. These will do things like go through provided assignment code, review background material, review big concepts, and/or do exercises. You can register for particular tutorial sections if you want to save a seat at a particular time, but note that you do not need to register in a tutorial section.
CPSC 340/532M vs. CPSC 440/540:
CPSC 340 and CPSC 440 are roughly structured as one full-year course. CPSC 340 (which is occasionally listed as CPSC 532M for graduate students) covers more data mining methods
and the methods that are most widely-used in applications of machine learning
while CPSC 440 (listed as CPSC 540 for graduate students) focuses on probabilistic methods which appear in more niche applications.
It is strongly recommended that you take CPSC 340 first, as it covers the most fundamental ideas as well as the most common and practically-useful techniques.
In 440 it will be assumed that you are familiar with all the material in the current offering of CPSC 340,
and note that online machine learning courses (and courses from many other universities)
are not an adequate replacement for CPSC 340 (they typically have more overlap with our applied machine learning course, CPSC 330).
Prerequisites:
- CPSC 320: Intermediate Algorithm Design and Analysis. Note that this course itself has the following prerequisites:
- Basic algorithms and data structures (CPSC 221 or EECE 320).
- CPSC 340: Machine Learning and Data Mining. Note that this course itself has the following additional prerequisites:
- Linear algebra (one of MATH 152, 221, or 223).
- Probability (one of STAT 241, STAT 251, ECON 325, ECON 327, MATH 302, STAT 302, or MATH 318).
- Multivariate calculus (one of MATH 200, 217, 226, 253, or 263).
Undergraduate students will not be able to take the class without these prerequisites.
Graduate students may be asked to show how they satisfy prerequisites.
Textbook: There is no textbook for the course, but the textbook with the most extensive coverage of many of the course's topics is Kevin Murphy's Machine Learning: A Probabilistic Perspective (MLAPP, 1st Edition). This book can be purchased from Amazon,
is on reserve in the CS Reading Room (ICCS 262), and can be accessed through the library here. Optional readings will be given out of this textbook, in addition to other free online resources.
Timetable
| Date |
Lecture Slides |
Related Readings and Links |
Homework |
| Mon Jan 10 |
Syllabus
Binary Density Estimation |
ML vs. Stats, 3 Cultures of ML
Mathematics for Machine Learning, Essence of Linear Algebra |
Assignment 1 (a1.zip, a1.tex) |
| Wed Jan 12 |
Bernoulli Distribution |
MLAPP 2.1-3 |
|
| Fri Jan 14 |
MAP Estimation |
MLAPP 3.1-3, Probability Primer |
|
| Mon Jan 17 |
Generative Classifiers |
MLAPP 3.5 |
|
| Wed Jan 19 |
Discriminative Classifiers |
MLAPP 8.1-3, 8.6 |
|
| Fri Jan 21 |
Neural Networks |
MLAPP 16.5 |
Assignment 1 Due |
| Mon Jan 24 |
Double Descent Curves |
|
Assignment 2 (a2.zip, a2.tex) |
| Wed Jan 26 |
Automatic Differentiation |
|
|
| Fri Jan 28 |
Convolutional Neural Networks |
|
|
| Mon Jan 31 |
Autoencoders |
MLAPP 28.3 |
|
| Wed Feb 2 |
Fully-Convolutional Networks |
|
|
| Fri Feb 4 |
Monte Carlo Approximation |
MLAPP 2.7, 23.1 |
|
| Mon Feb 7 |
Conjugate Priors |
MLAPP 3.4, 5.4 |
|
| Wed Feb 9 |
Bayesian Learning |
MLAPP 5.1-5.3 |
|
| Fri Feb 11 |
Empirical Bayes |
MLAPP 5.5-6 |
Assignment 2 due |
| Mon Feb 14 |
Multi-Class Classification |
MLAPP 5.7 |
Assignment 3 (a3.zip, a3.tex) |
| Wed Feb 16 |
What do we learn? |
|
|
| Fri Feb 18 |
Recurrent Neural Networks |
|
|
| Fri Feb 25 |
Bonus Lecture: Faster Algorithms for Deep Learning? |
|
|
| Mon Feb 28 |
Long Short Term Memory |
|
|
| Wed Mar 2 |
Attention and Transformers |
|
|
| Fri Mar 4 |
Univariate Gaussian |
MLAPP 2.4, 23.2 |
|
| Mon Mar 7 |
Multivariate Gaussian (Motivation)
Multivariate Gaussian (Details) |
MLAPP 2.5-6, 4.3-4 |
|
| Wed Mar 9 |
Learning Gaussians |
MLAPP 4.1-2, 4.5-6, 7.1-3, 7.5 |
|
| Fri Mar 11 |
Bayesian Linear Regression |
MLAPP 7.6, 8.4, 23.3-4 |
Assignment 3 due |
| Mon Mar 14 |
End to End Learning |
|
Assignment 4 (a4.zip, a4.tex) |
| Wed Mar 16 |
Exponential Family
Markov Chains |
MLAPP 9.1-3
MLAPP 17.-2 |
|
| Fri Mar 18 |
Learning Markov Chains |
MLAPP 17.1-2 |
|
| Mon Mar 21 |
Message Passing |
MLAPP 17.4 |
|
| Wed Mar 23 |
Markov Chain Monte Carlo |
MLAPP 24.1-6 |
|
| Fri Mar 25 |
Directed Acyclic Graphical Models |
MLAPP 10.1-2, 10.5 |
|
| Mon Mar 28 |
Learning Graphical Models |
MLAPP 10.3-4, 19.1-4 |
|
| Wed Mar 30 |
Log-Linear Models |
MLAPP 19.5-6 |
|
| Fri Apr 1 |
Mixture Models |
MLAPP 11.1-3 |
Assignment 4 due |
| Mon Apr 4 |
EM and KDE |
MLAPP 11.4, 14.7 |
|
| Wed Apr 6 |
HMMs and RBMs |
MLAPP 17.2, 17.5, 27.7, 28.2, 28.4 |
|
| Fri Apr 8 |
Topic Models and Variational Inference |
MLAPP 27.3-4, 21.1-5 |
|
Notes: These are additional notes mentioned in the lectures and homeworks:
Related Courses: Besides CPSC 340, other closely-related courses available at UBC include 500-level classes taught by Frank Wood, Leonid Sigal, Helge Rhodin, Kwang Moo Yi, Danica Sutherland, Jeff Clune, and Mijung Park. Related courses from other departments include
EECE 360/592, EOSC 510/550, STAT 305/306/406/460/461, and many 500-level STAT courses.
There is some discussion of how 340/440 relate to some of the undergraduate STAT classes written by a former student (Geoff Roeder) here, although it's getting a bit out of date (let me know if you have taken these courses recently and want to offer an update perspective).
Some related courses that have online notes are:
A YouTube playlist covering in detail many of the core topics in the course:
Mark Schmidt > Courses > CPSC 440
