OWA-Adam: Accelerating Convergence through Ordered Weighted Averaging in Adaptive Optimization

Abstract

This paper introduces OWA-Adam, a novel optimization algorithm addressing the challenges of efficient and adaptive optimization in deep learning. By incorporating the advantages of adaptive learning rate algorithms, gradient moment estimators, and ordered weighted average parameter updates, OWA-Adam aims to enhance convergence speed and generalization performance in deep learning models. Unlike traditional Adam which uses fixed exponential decay for moment estimation, OWA-Adam employs adaptive gradient aggregation strategies that consider the relative importance of historical gradients. Through comprehensive experimental validation across multiple independent trials, we demonstrate that OWA-Adam with exponential decay weighting achieves faster convergence compared to standard Adam while maintaining identical final performance across all evaluation metrics. Statistical significance testing across 10 independent trials confirms the robustness and reliability of these improvements. These findings underscore the potential of OWA-Adam to significantly improve the training process and overall performance of deep learning models. The implications of the proposed algorithm extend to both the research community and academia, offering advancements in optimization techniques for deep learning.