COMP 551: Applied Machine Learning

Winter 2021, delivered though McGill's MyCourses

Instructor: Reihaneh Rabbany

Teaching Assistants [click to expand the list]

Tianzi Yang [tianzi.yang]: Head TA

Tianyu Shi [tianyu.shi3]: Quiz TA

Nishant Mishra [nishant.mishra]: Group TA

Haque Ishfaq [haque.ishfaq]

Safa Alver [safa.alver]

Manoj Venkatesan [manoj.venkatesan]

Yan Miao [yan.miao]

Sumana Basu [sumana.basu]

Contact: comp551mcgill@gmail.com
please make sure to use this email to receive a timely response

Click here to expand all nested fields

Overview

Prerequisites [click to expand the list]

This course requires programming skills (python) and basic knowledge of probabilities, calculus and linear algebra. For more information see the course prerequisites and restrictions at McGill's webpage.

Textbooks

[Bishop] Pattern Recognition and Machine Learning by Christopher Bishop (2007)
[Goodfellow] Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville (2016)
[Murphy] Machine Learning: A Probabilistic Perspective by Kevin Murphy (2012)

Chapters from these three books are cited as optional reference materials for the slides.
There are several other related references. [click to expand the list]

The Elements of Statistical Learning: Data Mining, Inference, and Prediction by Trevor Hastie, Robert Tibshirani and Jerome Friedman (2009)

Information Theory, Inference, and Learning Algorithms, by David MacKay (2003)

Bayesian Reasoning and Machine Learning, by David Barber (2012).

Understanding Machine Learning: From Theory to Algorithms, by Shai Shalev-Shwartz and Shai Ben-David (2014)

Foundations of Machine Learning, by Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar (2018)

Dive into Deep Learning, by Aston Zhang, Zachary Lipton, Mu Li, and Alexander J. Smola (2019)

Mathematics for Machine Learning, by Marc Peter Deisenroth, A Aldo Faisal, and Cheng Soon Ong (2019)

A Course in Machine Learning, by Hal Daume III (2017)

Hands-on Machine Learning with Scikit-Learn and TensorFlow, by Aurelien Geron (2017)

Machine Learning, by Tom Mitchell (1997)

Introduction to Data Mining, by Pang-Ning Tan, Michael Steinbach, Anuj Karpatne, and Vipin Kumar (2020)

Machine Learning, Dynamical Systems and Control, by Steven L. Brunton and J. Nathan Kutz (2019)

Probabilistic Machine Learning: An Introduction, by Kevin P. Murphy (2021)

Schedule

Mon., Jan. 11: Syllabus

Wed., Jan. 13: Introduction

Mon., Jan. 18: Nearest Neighbours

Wed., Jan. 20: Decision Trees

covered slides: 1-12.3
Note: last lecture before the Add/Drop deadline

Mon., Jan. 25: Decision Trees continued & Core ML concepts

Core ML concepts slides: 1-3.3

Wed., Jan. 27: Core ML concepts continued

Core ML concepts slides up to the curse of dimensionality: slide 5.1

Mon., Feb. 1: Core ML concepts continued & Maximum likelihood and Bayesian Reasoning

Bayesian Reasoning up to posterior predictive, slide 5.1

Wed., Feb. 3: Bayesian Reasoning continued & Naive Bayes

Bayesian Reasoning from slide 5.1 & Naive Bayes up to until slide 8.2

Mon., Feb. 8: Naive Bayes continued

Wed., Feb. 10: Linear regression

Mon., Feb. 15: Logistic and softmax regression

Wed., Feb. 17: Gradient descent methods

until slide 6.5: MiniBatch SGD

Mon., Feb. 22: Gradient descent methods continued & Regularization

Wed., Feb. 24: Regularization continued with Bias-variance decomposition

~~Mon., Mar. 1~~

Note: Study break, No class [no quiz], see McGill's Academic calendar

~~Wed., Mar. 3~~

Note: Study break, No class [no quiz], see McGill's Academic calendar

Mon., Mar. 8: Multilayer Perceptrons

Wed., Mar. 10: Multilayer Perceptrons & Gradient computation and automatic differentiation

until slide 4.2

Mon., Mar. 15: Gradient computation and automatic differentiation

Wed., Mar. 17: Gradient computation and automatic differentiation Cont.

Mon., Mar. 22: Convolutional neural networks

Wed., Mar. 24: Support vector machines

Mon., Mar. 29: Bagging & Boosting

Wed., Mar. 31: Unsupervised learning

~~Mon., Apr. 5~~

Note: Easter Monday, No class [no quiz], see McGill's Academic calendar

Wed., Apr. 7: Dimensionality reduction

Mon., Apr. 12: Learning with graphs

Wed., Apr. 14: Ethics in ML, A guest lecture by Abhishek Gupta

Abhishek Gupta is the founder of Montreal AI Ethics Institute

Thr., Apr. 15: RL in practice, A guest lecture by by Marc Bellemare

Marc Bellemare leads the reinforcement learning efforts of the Google Research team in Montreal
Note: Makeup Day, see McGill's Academic calendar

Tutorials

Fri., Jan. 15: Probability and Linear Algebra

By Safa Alver, slides for linear algebra and probability, see Mycourses for the recording

Tue., Jan. 19: Python / NumPy

By Manoj

Mid Feb.: Scikit-Learn

By Nishant

Early Mar.: Pytorch

by Sumana

Outline

*** quick note ***

Please note that this list of topics is tentative, we might add/drop some topics or change the order as the course progresses. The slides will be posted here right after the corresponding lecture or sooner when possible. All the course materials (slides, codes, etc.) are adapted based on the last year's offering of the same course by Prof. Siamak Ravanbakhsh. You can look ahead into the materials, by checking that outline here.

Introduction

slides

Nearest Neighbours

slides, notebook (Colab), reference: chapter 1 [Murphy]

Classification and regression trees

slides, notebook (Colab), reference: 16.1-16.2.6 [Murphy], 14.4 [Bishop]

Core concepts

slides, notebook for model selection (Colab), notebook for curse of dimensionality (Colab)

Maximum likelihood and Bayesian Reasoning

slides, notebook (Colab), reference: 2-2.3 [Bishop], 3-3.5 [Murphy]

Naive Bayes

slides, notebook (Colab), reference: 3.5-3.5.4 [Murphy]

Linear regression

slides, notebook (Colab), reference: 7-7.3.3 [Murphy], 3-3.1.2 [Bishop]

Logistic and softmax regression

slides, notebook (Colab), reference: 8.1-8.3.3 [Murphy], 4.1-4.1.3 + 4.3-4.3.3 [Bishop]

Gradient descent methods

slides, notebook (Colab), reference: 8.3.2 [Murphy] and this overview by S. Ruder (in pdf )

Regularization

slides, notebook (Colab), reference: 3.1.4-3.3 [Bishop]

Perceptrons & Multilayer Perceptrons

slides, Perceptrons Colab, MLP demo, reference: 4.1.1-4.1.3 + 4.1.7 [Bishop], 6-6.5 + parts of 7 [Goodfellow]

Gradient computation and automatic differentiation

slides, notebook (Colab), reference: 6.5 + 8.2 [Goodfellow], blog post, visualization

Convolutional neural networks

slides, reference: 9 [Goodfellow], blog post, optional reading

Linear support vector machines

slides, notebook (Colab), reference: 4.1.1-4.1.3 + 4.1.7 + 7.1-7.1.4 excluding kernels [Bishop]

Bagging & Boosting

slides, reference: 3.2 [Bishop], demos for Bias-Variance Tradeoff, Gradient Boosting explanation, and Interactive playground

Unsupervised learning

slides, notebook (Colab), reference: 25.5 [Murphy] and 9.1 [Bishop], demos for K-Means and DB-SCAN

Dimensionality reduction

slides, notebook (Colab), reference: 12.2 [Murphy], 12.1 [Bishop], demo

Learning with graphs

slides,

Frontiers

Evaluation

Regular Practice Quizzes [20%]

One per lecture to check the key concepts discussed in the last lecture

Timed to be done in 1 hour after starting the quiz

Available until the start of the next lecture

Four lectures don't have practice quizzes and instead we have the checkpoint quizzes

From the two subsequent quizzes, the best one will be considered for your final grade

Starts from January 13th

The first 2 practice quizzes check the prerequisites for the course and have a different setting than the regular practice quizzes. In particular, unlimited attempts are allowed and due date is set to after add/drop instead of the next lecture

Knowledge Checkpoint Quizzes [30%]

One per month from all the topics discussed up to the date of the quiz

Timed to be done in 3 hours, within a 3 days availability period

Mark the due dates: Jan 21-23rd [3%], Feb 12-14th [7%], March 12-14th [10%], April 15-17th [10%]; timing (12:01 am to 11:59 pm)

Hands-on Mini-Projects [50%]

Four programming assignments to be done in groups of three*, *no exception to this given the grading load on TAs

Groups can stay the same between projects, you can also regroup when needed

All group members receive the same mark unless there are major complains on not contributing, responding, etc. from group-mates, which will be resolved in a case by case basis. If a significant difficulty/conflict arises, please send an email to the course email, cc the group-TA and put 'Group-TA' in the title

Mark the due dates: Feb 5th [10%], Feb 28th [15%], March 25th [15%], April 25th [10%]

Work submitted for evaluation as part of this course may be checked with text-matching software within myCourses

Late submission policy

All due dates are 11:59 pm in Montreal, unless specified otherwise [e.g. check the due dates for quizzes].
No make-up quizzes will be given.
For mini-projects, 2^k% percent will be deducted per k days of delay.
If you experience barriers (including a covid related issue) to learning in this course, submitting the projects, etc., please do not hesitate to discuss them with me directly, and please make sure to put "551 special" in the header to make sure I see your email [for general course correspondence, please use the course email: comp551mcgill@gmail.com]. As a point of reference, you can reach the Office for Students with Disabilities at 514-398-6009.