Course outline

See this page for help on Anaconda, Jupyter, and Python.

Part One

Linear Regression

Goals

• Learn the mathematics of Linear Regression (ordinary least squares) using Linear Algebra

  • Simple Linear Regression
  • Linear Regression - Geometry
  • Multivariate Regression - Introduction
    • Multivariate Regression - Calculus
  • Multivariate Regression - Geometry
  • Multivariate Regression - Centered Coordinates

• Lab:

  • get a working installation of anaconda, python, and jupyter on your computer.
  • a basic introduction to working with the Jupyter notebook
  • fundamentals of Python
  • calculations and plotting examples of linear regression

References

• Linear Regression html pdf
• Linear Regression lab - includes datafiles and ipynb file. zip tgz
• See also the Lab Resources page for help.

Gradient Descent

Goals:

• Learn the basic theory of gradient descent and how it is applied to find maxima and minima of functions

• Apply Gradient Descent to some specific examples.

  • Gradient Descent
  • Animation

References

• Gradient Descent lab - zip

Probability

Goals:

• Get an introduction to the key ideas from probability that play a role in machine learning.
• Learn about mean, variance, independence, conditional probability, and Bayes theorem.
• Try out the Naive Bayes classification method

• Introduce the ideas of statistical models and maximum likelihood.

• Probability Basics
• Conditional Probability 1
• Conditional Probability 2
• Probability - Independence
• Probability - Random Variables 1
• Probability - Random Variables 2
• Probability - Expectation and Variance
• Probability - Statistical Models

References

• Probability Notes html pdf
• Naive Bayes Notes html pdf
• Naive Bayes Lab - includes datafiles and ipynb file. zip tgz

Logistic Regression

Goals:

• Understand the statistical model underlying logistic regression
• See how the ideas of likelihood and gradient descent combine to solve the logistic regression problem
• Do some sample computations to see Logistic Regression in action
• Generalize binary logistic regression to multi-class logistic regression
  • Logistic Regression

References

• Logistic Regression lab - zip

Principal Component Analysis

Goals:

• Learn about how the covariance matrix encodes variation in linear combinations of features
• Learn the correlation coefficient
• Understand principal components and their relationship to eigenvectors of the covariance matrix
• See how to use principal components for dimension reduction.

• Do some examples.

• Variance, Covariance, Correlation, and the Covariance Matrix
• Scores
• Geometry of Scores
• Principal Directions - First Look
• Finding the Principal Directions
• The Spectral Theorem
• Dimension Reduction
• Loadings

References

• Principal Component Analysis html pdf
• PCA Lab – includes notebook(s) and data zip tgz

Bayesian Regression

Goals:

• Learn the process of Bayesian inference (see also the notes on probability above).
  • Intro to Bayesian Inference
  • Bayesian Coin Flipping
• Understand over-fitting in linear regression
• Study Bayesian linear regression
• Understand the ideas of linear discriminant analysis
  • Bayesian Linear Regression
  • Linear Discriminant Analysis
  • Dimensionality Reduction via LDA

References

• Bayesian Regression Lab – includes notebook(s) and data zip tgz

Support Vector Machines

Goals:

• Learn the ideas behind support vector machine classifiers

  • Introduction
  • Optimal Margins

• Understand the relationship between convex hulls, supporting hyperplanes, and support vector machines

  • Convexity, Convex Hulls, and Supporting Hyperplanes
• Formulate the convex optimization problem yielding the optimal margin classifier
  • Closest Points and Optimal Margins 1
  • Closest Points and Optimal Margins 2
  • Formulating the Optimization Problem
• Learn the sequential minimum optimization algorithm
  • The SMO Algorithm
• Further ideas
  • Kernels, inseparable sets, multiclass classification

References:

• Notes on support vector machines html pdf
• Support Vector Machines Lab (includes jupyter notebook and data files) zip tgz

Neural Networks

Goals:

• Learn the ideas of neural networks and understand forward-propagation and back-propagation
• Derive the back-propagation formula
• Learn the mechanisms of basic convolutional neural networks
  • Basic Neural Networks
  • Convolutional Neural Networks

References

• Neural Networks Lab – includes notebook(s) and data zip tgz