Inertial sensors are integral components in numerous applications, powering crucial features in robotics and our daily lives. In recent years, deep learning has significantly advanced inertial sensing performance and robustness. Deep-learning techniques are used in different domains and platforms to enhance network performance, but no common benchmark is available. The latter is critical for fair comparison and evaluation in a standardized framework as well as development in the field. To fill this gap, we define and thoroughly analyze 13 data-driven techniques for improving neural inertial regression networks. A focus is placed on three aspects of neural networks: network architecture, data augmentation, and data preprocessing. Extensive experiments were made across six diverse datasets that were collected from various platforms including quadrotors, doors, pedestrians, and mobile robots. In total, over 1079 minutes of inertial data sampled between 120-200Hz were analyzed. Our results demonstrate that data augmentation through rotation and noise addition consistently yields the most significant improvements. Moreover, this study outlines benchmarking strategies for enhancing neural inertial regression networks.