BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Laboratoire de Mathématiques de Versailles - ECPv6.15.18//NONSGML v1.0//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
X-WR-CALNAME:Laboratoire de Mathématiques de Versailles
X-ORIGINAL-URL:https://lmv.math.cnrs.fr
X-WR-CALDESC:Évènements pour Laboratoire de Mathématiques de Versailles
REFRESH-INTERVAL;VALUE=DURATION:PT1H
X-Robots-Tag:noindex
X-PUBLISHED-TTL:PT1H
BEGIN:VTIMEZONE
TZID:Europe/Paris
BEGIN:DAYLIGHT
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
TZNAME:CEST
DTSTART:20240331T010000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
TZNAME:CET
DTSTART:20241027T010000
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
TZNAME:CEST
DTSTART:20250330T010000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
TZNAME:CET
DTSTART:20251026T010000
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
TZNAME:CEST
DTSTART:20260329T010000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
TZNAME:CET
DTSTART:20261025T010000
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTART;TZID=Europe/Paris:20250214T113000
DTEND;TZID=Europe/Paris:20250214T123000
DTSTAMP:20260406T224107
CREATED:20240904T164618Z
LAST-MODIFIED:20250207T112138Z
UID:13098-1739532600-1739536200@lmv.math.cnrs.fr
SUMMARY:PS : Umut Simsekli (INRIA et ENS Paris) : Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGD
DESCRIPTION:Neural network compression has been an increasingly important subject\, not only due to its practical relevance\, but also due to its theoretical implications\, as there is an explicit connection between compressibility and generalization error. In this talk\, I will present a simple modification for SGD\, such that the outputs of the algorithm will be provably compressible without making any nontrivial assumptions. We will consider a one-hidden-layer neural network trained with SGD\, and show that if we inject additive heavy-tailed noise to the iterates at each iteration\, for any compression rate\, there exists a level of overparametrization such that the output of the algorithm will be compressible with high probability. To achieve this result\, we make two main technical contributions: (i) we prove a “propagation of chaos” result for a class of heavy-tailed stochastic differential equations\, and (ii) we derive error estimates for their Euler discretization. Our experiments suggest that the proposed approach not only achieves increased compressibility with various models and datasets\, but also leads to robust test performance under pruning\, even in more realistic architectures that lie beyond our theoretical setting. The talk is based on the following article: https://arxiv.org/pdf/2306.08125.pdf
URL:https://lmv.math.cnrs.fr/evenenement/ps-umut-simsekli-inria-et-ens-paris/
LOCATION:Bâtiment Fermat\, salle 4205
CATEGORIES:Séminaire PS
END:VEVENT
END:VCALENDAR