A NOVEL DECAY STEP SIZE FOR STOCHASTIC GRADIENT DESCENT

Abstract

In this paper, we propose a modied version of the p1 t step size and demonstrate its convergence rate of O( lpn T T ) for smooth non-convex functions without the Polyak-ojasiewicz con- dition. The numerical results show that new step size leads to improved accuracy, with enhancements of 0:5%, 1:4%, and 1:4% observed in image classication tasks on the FashionMNIST, CI- FAR10, and CIFAR100 datasets, respectively, compared to the p1 t step size.