Adapting the spectral viewpoint suggested in Ledoux (2012) in the context of symmetric Markov diffusion generators and recently exploited in the non-diffusive setup of a Poisson random measure by Döbler and Peccati (2017), we investigate the fourth moment condition for discrete multiple integrals with respect to general, i.e.\ non-symmetric and non-homogeneous, Rademacher sequences and show that, in this situation, the fourth moment alone does not govern the asymptotic normality. Indeed, here one also has to take into consideration the maximal influence of the corresponding kernel functions. In particular, we show that there is no exact fourth moment theorem for discrete multiple integrals of order $m\geq2$ with respect to a symmetric Rademacher sequence. This behavior, which is in contrast to the Gaussian (see Nualart and Peccati (2005)) and Poisson (see Döbler and Peccati (2017)) situation, closely resembles the conditions for asymptotic normality of degenerate, non-symmetric $U$-statistics from the classical paper by de Jong (1990).