Bolte, JérômeORCID: https://orcid.org/0000-0002-1676-8407, Glaudin, Lilian, Pauwels, EdouardORCID: https://orcid.org/0000-0002-8180-075X and Serrurier, Matthieu (2023) The backtrack Hölder gradient method with application to min-max and min-min problems. Open Journal of Mathematical Optimization, vol.4 (n°8).

Abstract

We present a new algorithm to solve min-max or min-min problems out of the convex world. We use rigidity assumptions, ubiquitous in learning, making our method – the backtrack Hölder algorithm applicable to many optimization problems. Our approach takes advantage of hidden regularity properties and allows us, in particular, to devise a simple algorithm of ridge type. An original feature of our method is to come with automatic step size adaptation which departs from the usual overly cautious backtracking methods. In a general framework, we provide convergence theoretical guarantees and rates. We apply our findings on simple Generative Adversarial Network (GAN) problems obtaining promising numerical results. It is worthwhile mentioning that a byproduct of our approach is a simple recipe for general Hölderian backtracking optimization.