Neural networks (NNs) for multiple hardware impairments mitigation of a realistic direct conversion transmitter are impractical due to high computational complexity. We propose two methods to reduce complexity without significant performance penalty. We first propose a novel attention residual learning NN, referred to as attention residual real-valued time-delay neural network (ARDEN), where trainable neuron-wise shortcut connections between the input and output layers allow to keep the attention always active. Furthermore, we implement a NN pruning algorithm that gradually removes connections corresponding to minimal weight magnitudes in each layer. Simulation and experimental results show that ARDEN with pruning achieves better performance for compensating frequency-dependent quadrature imbalance and power amplifier nonlinearity than other NN-based and Volterra-based models, while requiring less or similar complexity.