The generalization error bound for a stochastic gradient descent family via a Gaussian approximation method - Chen, Hao; Mo, Zhanfeng; Yang, Zhouwang (Zielona Góra: Uniwersytet Zielonogórski)
Recent works have developed model complexity based and algorithm based generalization error bounds to explain how stochastic gradient descent (SGD) methods help over-parameterized models generalize better. However, previous works are limited by their scope of analysis and fail to provide comprehensive explanations. In this paper, we propose a novel Gaussian approximation framework to establish generalization error bounds for the U-SGD family, which is a class of SGD with asymptotically unbiased and uniformly bounded gradient noise. We study U-SGD dynamics, and we show both theoretically and numerically that the limiting model parameter distribution tends to be Gaussian, even when the original gradient noise is non-Gaussian.
Keywords: stochastic gradient descent; Gaussian approximation; KL-divergence; generalization bounds