The paper presents a novel type of capsule network (CAP) that uses custom-defined neural network (NN) layers for blind classification of digitally modulated signals using their in-phase/quadrature (I/Q) components. The custom NN layers of the CAP are inspired by cyclostationary signal processing (CSP) techniques and implement feature extraction capabilities that are akin to the calculation of the cyclic cumulant (CC) features employed in conventional CSP-based approaches to blind modulation classification and signal identification. The classification performance and the generalization abilities of the proposed CAP are tested using two distinct datasets that contain similar classes of digitally modulated signals but that have been generated independently, and numerical results obtained reveal that the proposed CAP with novel NN feature extraction layers achieves high classification accuracy while also outperforming alternative deep learning (DL)-based approaches for signal classification in terms of both classification accuracy and generalization abilities.