Events Mini Calendar

Go to previous month Go to next month
  S M T W R F S
W9 25 26 27 28 29 1 2
W10 3 4 5 6 7 8 9
W11 10 11 12 13 14 15 16
W12 17 18 19 20 21 22 23
W13 24 25 26 27 28 29 30
W14 31 1 2 3 4 5 6


Today is:
Thu, Mar 28, 2024


MSU Events Calendar

 
 Choose Calendar view:   Day   Week   Month     Search   Filter

2:30pm
to
3:30pm
Math Seminar Series  (Conferences / Seminars / Lectures)

Applied Mathematics

Mikhail Belkin , University of California, San Diego, will be speaking.

The presentation is titled "A theory of optimization and transition to linearity in deep learning."

Contact Olga Turanova (turanova@msu.edu) for more information.

The success of deep learning is due, to a large extent, to the remarkable effectiveness of gradient-based optimization methods applied to large neural networks. In this talk I will discuss some general mathematical principles allowing for efficient optimization in over-parameterized non-linear systems, a setting that includes deep neural networks. Remarkably, it seems that optimization of such systems is "easy". In particular, optimization problems corresponding to these systems are not convex, even locally, but instead satisfy locally the Polyak-Lojasiewicz (PL) condition allowing for efficient optimization by gradient descent or SGD. We connect the PL condition of these systems to the condition number associated to the tangent kernel and develop a non-linear theory parallel to classical analyses of over-parameterized linear equations. In a related but conceptually separate development, I will discuss a new perspective on the remarkable recently discovered phenomenon of transition to linearity (constancy of NTK) in certain classes of large neural networks. I will show how this transition to linearity results from the scaling of the Hessian with the size of the network. Joint work with Chaoyue Liu and Libin Zhu

more information...


Location: Online via Zoom [map]
Price: free
Sponsor: Department of Mathematics
Contact: Department of Mathematics
phone pic (517) 353-0844