In this paper, we focus on the joint impairments compensation and symbol detection problem in communication systems. First, we introduce a new multi-layer channel model that represents the underlying physics of multiple impairments in communication systems. This model is composed of widely linear parametric layers that describe the input-output relationship of the front-end impairments and channel effects. Using this particular model, we show that the joint compensation and zero-forcing detection problem can be solved by a particular feedforward network called PhyCOM. Because of the small number of network parameters, a PhyCOM network can be trained efficiently using sophisticated optimization algorithms and a limited number of pilot symbols. Numerical examples are provided to demonstrate the effectiveness of PhyCOM networks with communication systems corrupted by transmitter and receiver IQ imbalances, carrier frequency offset, finite impulse response channels, and transmitter and receiver phase noise distortions. Compared to conventional digital signal processing approaches, simulation results show that the proposed technique is much more flexible and offers better statistical performance both in terms of MSE and SER with a moderate increase of the computation complexity.