Direction-of-Arrival (DoA) estimation assisted with an Intelligent Reflecting Surface (IRS) is crucial for various wireless applications, especially in challenging Non-Line-of-Sight (NLoS) environments. This paper presents a novel neural network-based architecture to address this challenge. The key innovation is the introduction of a dedicated, learnable IRS layer integrated within a carefully designed end-to-end system established upon the physical and geometrical basis of the problem. Unlike conventional neural network layers, this specific one incorporates block diagonal sinusoidal weight constraints, where the phase arguments of these sinusoids are learned during training to directly emulate the phase shifts of the IRS elements. This allows the end-to-end system to optimize the IRS configuration for enhanced DoA estimation, eliminating the need for separate IRS optimization algorithms. Moreover, different DoA regression networks, including a proposed structure, are presented and examined. Numerical simulations, conducted under various conditions and noise levels, where controlled coherent multi-path components are introduced due to the presence of the IRS, demonstrate the superior performance of the novel end-to-end system compared to others and highlight its potential to significantly improve the accuracy of DoA estimation in complex IRS-assisted wireless systems. Besides, corresponding computational complexities of different approaches are also compared.