The worm gearbox is a high-speed transmission system that plays a vital role in various industries. Therefore it becomes necessary to develop a robust fault diagnosis scheme for worm gearbox. Due to advancements in sensor technology, researchers from academia and industries prefer deep learning models for fault diagnosis purposes. The optimal selection of hyperparameters (HPs) of deep learning models plays a significant role in stable performance. Existing methods mainly focused on manual tunning of these parameters, which is a troublesome process and sometimes leads to inaccurate results. Thus, exploring more sophisticated methods to optimize the HPs automatically is important. In this work, a novel optimization, i.e. amended gorilla troop optimization (AGTO), has been proposed to make the convolutional neural network (CNN) adaptive for extracting the features to identify the worm gearbox defects. Initially, the vibration and acoustic signals are converted into 2D images by the Morlet wavelet function. Then, the initial model of CNN is developed by setting hyperparameters. Further, the search space of each Hp is identified and optimized by the developed AGTO algorithm. The classification accuracy has been evaluated by AGTO-CNN, which is further validated by the confusion matrix. The performance of the developed model has also been compared with other models. The AGTO algorithm is examined on twenty-three classical benchmark functions and the Wilcoxon test which demonstrates the effectiveness and dominance of the developed optimization algorithm. The results obtained suggested that the AGTO-CNN has the highest diagnostic accuracy and is stable and robust while diagnosing the worm gearbox.