• NeurIPS 2024 (Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models)
• NeurIPS 2023 (Searching for Optimal Per-Coordinate Step-sizes with Multidimensional Backtracking)
• NeurIPS 2023 (Don't be so Monotone: Relaxing Stochastic Line Search in Over-Parameterized Models)
• ICLR 2023 (Noise is not the main factor behind the gap between SGD and Adam on transformers, but sign descent might be)
• JMLR 2022 (Let's Make Block Coordinate Descent Go Fast: Faster Greedy Rules, Message-Passing, Active-Set Complexity, and Superlinear Convergence)
• AI/Stats 2021 (Homeomorphic-Invariance of EM: Non-Asymptotic Convergence in KL Divergence for Exponential Families via Mirror Descent)
• arXiv 2020 (Adaptive Gradient Methods Converge Faster with Over-Parameterization (but you should do a line-search))
• AI/Stats 2020 (Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation)
• NeurIPS 2019 (Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates)
• AI/Stats 2019 (Fast and Faster Convergence of SGD for Over-Parameterized Models (and an Accelerated Perceptron))

• ICML 2023 (Target-based Surrogates for Stochastic Optimization)
• ICML 2023 (Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learning)
• CoLLAs 2022 (Improved Policy Optimization for Online Imitation Learning)
• ICML 2021 (Robust Asymmetric Learning in POMDPs)
• ICML 2021 (Tractable structured natural gradient descent using local parameterizations)
• ICML 2020 (Handling the Positive-Definite Constraint in the Bayesian Learning Rule)
• MLJ 2020 (Combining Bayesian Optimization and Lipschitz Optimization)
• BMVC 2019 (Does Your Model Know the Digit 6 is Not a Cat? A Less Biased Evaluation of "Outlier" Detectors)
• ICML 2019 (Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations)
• NeurIPS 2018 (SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient)

• Professional-quality face retouching
• Recognizing distinct objects in images
• Counting objects in images
• Outdoor image segmentation and depth estimation
• Artistic style transfer for images and videos
• Image restoration and inpainting
• Assessing malignance in mammography images
• Computed tomography muscle segmentation
• Heart motion abnormality detection
• Automatic brain tumor segmentation

• Training large language models
• Safe learning for self-driving cars
• Modeling kinematics of DNA strands
• Propagating ideas in social networks
• Product recommendation in social networks
• Acoustic waveform inversion for seismic imaging
• Predicting the effects of new combinations of interventions
• Modeling perturbations in intracellular flow cytometry
• Finding groups of correlated mutual funds
• Network models of gene expression data

Presentations

• ECML 2020 (Combining Bayesian Optimization and Lipschitz Optimization)
• AI/Stats 2020 (Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation)
• ICML 2020 (Handling the Positive-Definite Constraint in the Bayesian Learning Rule)
• NeurIPS 2019 (Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates)
• ISMP 2018 (Convergence Rate of Expectation Maximization)
• DIMACS Opt and ML 2018 ("Active-set complexity" of proximal-gradient: How long does it take to find the sparsity pattern?)
• Simons Institute 2017 (Let's Make Block Coordinate Descent Go Fast!)
• NeurIPS OptOpt 2016 (Do we need "Harmless" Bayesian Optimization and "First-Order" Bayesian Optimization)
• ECML 2016 (Linear Convergence under the Polyak-Lojasiewicz Inequality)
• ISMP 2015 (Advances in the Minimization of Finite Sums)

• Alberta 2024 (Understanding and improving the numerical optimization underlying modern machine learning)
• MPI + UCLA 2022 (Optimization Algorithms for Training Over-Parameterized Models)
• Vector Institute 2020 (Faster Algorithms for Deep Learning?)
• SIOPT 2017 (Converging on the Ultimate Algorithm for Minimizing Convex Sums)
• Northwestern 2016 (The Why and How of Releasing Applied Math Code)
• TAAI 2014 (Tractable Big Data and Big Models in Machine Learning)
• Kyoto 2014 (Opening up the black box)
• Paris-6 2013 (Linearly-Convergent Stochastic-Gradient Methods)
• INRA 2011 (Structure Learning in Undirected Graphical Models)
• NeurIPS OPT 2010 (Limited-Memory Quasi-Newton and Hessian-Free Newton Methods)

• NeurIPS 2019 (Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates)
• NeurIPS WiML 2019 (Newton-Laplace Updates for Block Coordinate Descent)
• AI/Stats 2019 (Fast and Faster Convergence of SGD for Over-Parameterized Models (and an Accelerated Perceptron))
• AI/Stats 2019 (Are we there yet? Manifold identification of gradient-related proximal methods)
• NeurIPS WiML and OPT 2017 (Linear Convergence and Support Vector Identification of Sequential Minimal Optimization)
• NeurIPS WiML 2017 (Let's Make Block Coordinate Descent Go Fast!)
• NeurIPS OPT 2017 (Convergence Rate of Expectation Maximization)
• NeurIPS OPT 2017 ("Active-set complexity" of proximal-gradient: How long does it take to find the sparsity pattern?)
• NeurIPS BayesOpt 2016 (Do we need "Harmless" Bayesian Optimization and "First-Order" Bayesian Optimization?)
• ECML 2016 (Linear Convergence of Gradient and Proximal-Gradient Methods under the Polyak-Lojasiewicz Condition)

• MPI + UCLA 2022 (Optimization Algorithms for Training Over-Parameterized Models)
• ICML 2020 (Handling the Positive-Definite Constraint in the Bayesian Learning Rule) )
• AI/Stats 2020 (Fast and Furious Convergence: Stochastic Second-Order Methods under Interpolation)
• NeurIPS 2019 (Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates )
• Simons Institute 2017 (Let's Make Block Coordinate Descent Go Fast)
• ICML 2015 (Coordinate Descent Converges Faster with the Gauss-Southwell Rule than Random Selection)
• NeurIPS OPT 2014 (Non-Uniform Stochastic Average Gradient for Training Conditional Random Fields)
• BIRS 2014 (Minimizing Finite Sums with the Stochastic Average Gradient)
• Boulder 2014 (Fast Non-Smooth and Big-Data Optimization)
• ICML 2013 (Block-Coordinate Frank-Wolfe Optimization)

Code

• matLearn (2016, machine learning in Matlab)
• SAG4CRF (2015, stochastic average gradient for conditional random fields)
• SAG (2013, L2-regularized logistic regression)
• thesis (2012, code from my thesis)
  • examples with simple regularizers
• batching (2011) (growing-batch optimization)
• UGM (updated 2011) (undirected graphical models)
• L1General (updated 2010, L1-regularized optimization)
  • examples
• minConf (2009, optimization with simple constraints)
  • examples with bound constraints
  • examples with simple constraints
• lasso (updated 2008, L1-regularized least squares)
• minFunc (updated 2012, differentiable optimization)
  • examples

Teaching and Research Groups