Bolte, Jérôme and Pauwels, Edouard (2019) Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning. Mathematical Programming. pp. 1-33. (In Press)

[thumbnail of wp_tse_1044.pdf]
Download (486kB) | Preview
Identification Number : 10.1007/s10107-020-01501-5


Modern problems in AI or in numerical analysis require nonsmooth approaches with a
exible calculus. We introduce generalized derivatives called conservative
fields for which we develop a calculus and provide representation formulas. Functions having a conservative field are called path differentiable: convex, concave,
Clarke regular and any semialgebraic Lipschitz continuous functions are path differentiable. Using Whitney stratification techniques for semialgebraic and definable
sets, our model provides variational formulas for nonsmooth automatic diffrentiation oracles, as for instance the famous backpropagation algorithm in deep learning.
Our differential model is applied to establish the convergence in values of nonsmooth stochastic gradient methods as they are implemented in practice.

Item Type: Article
Language: English
Date: 7 October 2019
Refereed: Yes
Place of Publication: Heidelberg
Uncontrolled Keywords: Deep Learning, Automatic differentiation, Backpropagation algorithm, Nonsmooth stochastic optimization, Defiable sets, o-minimal structures, Stochastic gradient, Clarke subdifferential, First order methods
Divisions: TSE-R (Toulouse)
Site: UT1
Date Deposited: 25 Mar 2021 10:11
Last Modified: 23 Jul 2021 13:06
OAI Identifier:
View Item


Downloads per month over past year