• NeurIPS 2024 (Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models)
• NeurIPS 2023 (Searching for Optimal Per-Coordinate Step-sizes with Multidimensional Backtracking)
• NeurIPS 2023 (Don't be so Monotone: Relaxing Stochastic Line Search in Over-Parameterized Models)
• ICLR 2023 (Noise is not the main factor behind the gap between SGD and Adam on transformers, but sign descent might be)
• JMLR 2022 (Let's Make Block Coordinate Descent Go Fast: Faster Greedy Rules, Message-Passing, Active-Set Complexity, and Superlinear Convergence)
• AI/Stats 2021 (Homeomorphic-Invariance of EM: Non-Asymptotic Convergence in KL Divergence for Exponential Families via Mirror Descent)
• arXiv 2020 (Adaptive Gradient Methods Converge Faster with Over-Parameterization (but you should do a line-search))
• AI/Stats 2020 (Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation)
• NeurIPS 2019 (Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates)
• AI/Stats 2019 (Fast and Faster Convergence of SGD for Over-Parameterized Models (and an Accelerated Perceptron))

• ICML 2023 (Target-based Surrogates for Stochastic Optimization)
• ICML 2023 (Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learning)
• CoLLAs 2022 (Improved Policy Optimization for Online Imitation Learning)
• ICML 2021 (Robust Asymmetric Learning in POMDPs)
• ICML 2021 (Tractable structured natural gradient descent using local parameterizations)
• ICML 2020 (Handling the Positive-Definite Constraint in the Bayesian Learning Rule)
• MLJ 2020 (Combining Bayesian Optimization and Lipschitz Optimization)
• BMVC 2019 (Does Your Model Know the Digit 6 is Not a Cat? A Less Biased Evaluation of "Outlier" Detectors)
• ICML 2019 (Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations)
• NeurIPS 2018 (SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient)

Applications

• Professional-quality face retouching
• Recognizing distinct objects in images
• Counting objects in images
• Outdoor image segmentation and depth estimation
• Artistic style transfer for images and videos
• Image restoration and inpainting
• Assessing malignance in mammography images
• Computed tomography muscle segmentation
• Heart motion abnormality detection
• Automatic brain tumor segmentation

• Training large language models
• Safe learning for self-driving cars
• Modeling kinematics of DNA strands
• Propagating ideas in social networks
• Product recommendation in social networks
• Acoustic waveform inversion for seismic imaging
• Predicting the effects of new combinations of interventions
• Modeling perturbations in intracellular flow cytometry
• Finding groups of correlated mutual funds
• Network models of gene expression data

Talks

• NYU 2025 (Why does Adam work so well for LLMs? And can we find optimal per-variable step sizes?)
• Alberta 2024 (Understanding and improving the numerical optimization underlying modern machine learning)
• MPI + UCLA 2022 (Optimization Algorithms for Training Over-Parameterized Models) - video
• Vector Institute 2020 (Faster Algorithms for Deep Learning?) - video
• SIOPT 2017 (Converging on the Ultimate Algorithm for Minimizing Convex Sums)
• Northwestern 2016 (The Why and How of Releasing Applied Math Code)
• ISMP 2015 (Advances in the Minimization of Finite Sums)
• TAAI 2014 (Tractable Big Data and Big Models in Machine Learning) - video
• Kyoto 2014 (Opening up the black box)
• Paris-6 2013 (Linearly-Convergent Stochastic-Gradient Methods)
• INRA 2011 (Structure Learning in Undirected Graphical Models)
• NeurIPS OPT 2010 (Limited-Memory Quasi-Newton and Hessian-Free Newton Methods for Non-Smooth Optimization)
• Thesis Defense 2010 (Graphical Model Structure Learning with L1-Regularization)
• IBM 2009 (Optimizing Costly Functions with Simple Constraints: A Limited-Memory Projected Quasi-Newton Algorithm)

Teaching and Tutorials

UBC Courses:

• CPSC 340 and 540: Machine Learning and Data Mining (Fall 2023, Fall 2022, Fall 2019, Fall 2018, Fall 2017, Fall 2016, Fall 2015).
• CPSC 440: Advanced Machine Learning (Winter 2022, Winter 2021, Winter 2020, Winter 2019, Winter 2018, Winter 2017, Winter 2016, Fall 2014).

• 5XX 2022 (Numerical Optimization for Machine Learning)
• 5XX 2020 (First-Order Optimization Algorithms for Machine Learning)
• SVAN 2016 (Stochastic Convex Optimization Methods in Machine Learning - videos)
• MLSS 2011 (Convex Optimization)

• DLRL 2022 (Introduction to Optimization -Theory and Practice)
• SIOPT 2017 (Stochastic Optimization for Machine Learning: Weakening the Assumptions)
• DLSS 2015 (Part 2: Non-Smooth, Non-Finite, and Non-Convex Optimization - video)
• DLSS 2015 (Part 1: Smooth, Finite, and Convex Optimization - video)
• ICML 2015 (Modern Convex Optimization Methods for Large-Scale Empirical Risk Minimization - video)
• ICASSP 2015 (Convex Optimization for Big Data)
• MLSS 2015 (Convex Optimization - video)
• ACML 2014 (Convex Optimization for Big Data)
• UBC 2009 (Linear Algebra)

Group

Current members:

• Curtis Fox (PhD)
• Dylan Green (MSc)
• Chen Fan (PhD)
• Donney Fan (PhD)
• Yunxiang Li (PhD)
• Alan Milligan (MSc)
• Amrutha Varshini Ramesh (PhD)
• Betty Shea (PhD)
• Robin Yadav (BSc)
• Malcolm Zhao (BSc)

• Mohamed Ahmed (PhD, now at Borealis AI)
• Saeid Allahdadian (Postdoc, now at Oracle)
• Reza Babanezhad (PhD, now at Samsung AI)
• Jacques Chen (BSc, now at Nuro)
• Ricky Chen (MSc, now at Meta AI)
• Benjamin Dubois-Taine (MSc, now PhD at ENS)
• Mehrdad Ghadiri (MSc, now Postdoc at MIT)
• Bingshan Hu (Postdoc, now Postdoc at UBC)
• Nicholas Ioannidis (BSc, now MSc at UBC)
• Hamed Karimi (Postdoc, now at Amazon)
• Raunak Kumar (BSc, now Microsoft)
• Frederik Kunstner (PhD, now at ENS)
• Lironne Kurzman (MSc)
• Issam Laradji (PhD, now at ServiceNow)
• Jonathan Wilder Lavington (PhD, now at Amazon)
• Wu Lin (PhD, now postdoc at Vector)
• Liam Madden (Postdoc)
• Si Yi (Cathy) Meng (MSc, now PhD at Cornell)
• Aaron Mishkin (MSc, now PhD at Stanford)
• Julie Nutini (PhD, now at Planet Labs)
• Michael Przystupa (M.Sc., now PhD at Alberta)
• Yasha Pushak (PhD, now at Oracle)
• Geoff Roeder (BSc, now PhD at Princeton)
• Alireza Shafaei (PhD, now CTO at Skylab)
• Jennifer (Xin Bei) She (B.Sc., now at DeepMind)
• Behrooz Sepehry (MSc, now at Amazon)
• Sharan Vaswani (PhD, now assistant professor at SFU)
• Alim Virani (BSc, now at Google)
• Shu Wang (BSc, now MSc at Oxford)
• Yihan (Joey) Zhou (MSc, now PhD at UT-Austin)
• Han (Alyssa) Zhang (BSc)
• Tianyue (Helen) Zhang (MSc, now PhD at MILA)
• Nasim Zolaktaf (PhD, now at Microsoft)

UBC Machine Learning Reading Group

Code