Loss landscape analysis for deep learning: A survey

Subjects: Mathematics >> Control and Optimization. submitted time 2021-11-29

梁若冰 刘波 孙越泓

Abstract： In the filed of machine learning and mathematical optimization, it is a challenge to mathematically explain optimality of loss function for deep learning. Loss function is high-dimensional, non-convex, and non-smooth. It was, however, observed that gradient descent could reach zero training loss of this highly non-convex function. Loss landscape analysis is critical to reveal reasons why deep networks are easily optimizable. We reviewed the advance on loss landscape analysis, such as landscape features (number and spatial distribution of local minima, connectivity between global optima, and global optimality of critical points), convergence of gradient descent, and visualization of loss landscape. This survey aimed to promote interpretable and reliable deep learning in critical applications. "

Peer Review Status:Awaiting Review

 Hits 10635  Downloads 988  Comment