Tang, Dahao, Yang, Nan, Deng, Yongkun, Zhang, Yuning, Sani, Abubakar Sadiq and Yuan, Dong (2024) Stability-driven CNN training with Lyapunov-based dynamic learning rate. In: Australasian Database Conference 2024, 28th Oct - 1st Nov 2024, Pullman Albert Park, Melbourne. Australasian Database Conference 2024, Melbourne, Australia. (In Press)

Preview

PDF (Accepted conference paper) 48299 SANI_Stability_Driven_CNN_Training_with_Lyapunov_Based_Dynamic_Learning_Rate_(AAM)_2024.pdf - Accepted Version Download (212kB) | Preview

Abstract

In recent years, Convolutional Neural Networks (CNNs) have become a cornerstone in computer vision tasks, yet their training stability remains a challenge, especially when using high learning rates or large datasets. Traditional optimization methods like Stochastic Gradient Descent (SGD) often suffer from oscillations and slow convergence, necessitating more robust techniques to ensure reliable training outcomes. This paper introduces a novel approach to enhancing the stability of CNNs during training by integrating control theory and Lyapunov Stability Analysis. We model the training process of CNNs as a dynamic control system and employ a Quadratic Lyapunov Function to assess and ensure its stability. The proposed framework enables dynamic adaptation of the learning rate, guided by real-time stability metrics, to prevent oscillations and improve convergence. We provide theoretical justifications, practical guidelines for hyper-parameter adaptation, and empirical results demonstrating the effectiveness of the approach in maintaining stability during training. Experiments were conducted on datasets CIFAR-10 comparing the performance of using traditional SGD and SGD-DLR (SGD with the dynamic learning rate) to confirm the advantages of our method.

Item Type:	Conference Proceedings
Title of Proceedings:	Australasian Database Conference 2024, 28th Oct - 1st Nov 2024, Pullman Albert Park, Melbourne
Uncontrolled Keywords:	convolutional neural networks, control theory, Lyapunov stability analysis, learning rate
Subjects:	Q Science > Q Science (General) Q Science > QA Mathematics Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Faculty / School / Research Centre / Research Group:	Faculty of Engineering & Science Faculty of Engineering & Science > School of Computing & Mathematical Sciences (CMS)
Related URLs:	Organisation
Last Modified:	16 Oct 2024 13:07
URI:	http://gala.gre.ac.uk/id/eprint/48299

Actions (login required)

View Item

Downloads