Diffractive deep neural network (D2NN), also referred to as reconfigurable intelligent metasurface based deep neural networks (Rb-DNNs) or stacked intelligent metasurfaces (SIMs) in the field of wireless communications, has emerged as a promising signal processing paradigm that enables computing-by-propagation. However, existing architectures are limited to implementing specific functions such as precoding and combining, while still relying on digital baseband modules for other essential tasks like modulation and detection. In this work, we propose a baseband-free end-to-end (BBF-E2E) wireless communication system where modulation, beamforming, and detection are jointly realized through the propagation of electromagnetic (EM) waves. The BBF-E2E system employs D2NNs at both the transmitter and the receiver, forming an autoencoder architecture optimized as a complex-valued neural network. The transmission coefficients of each metasurface layer are trained using the mini-batch stochastic gradient descent method to minimize the cross-entropy loss. To reduce computational complexity during diffraction calculation, the angular spectrum method (ASM) is adopted in place of the Rayleigh-Sommerfeld formula. Extensive simulations demonstrate that BBF-E2E achieves robust symbol transmission under challenging channel conditions with significantly reduced hardware requirements. In particular, the proposed system matches the performance of a conventional multi-antenna system with 81 RF chains while requiring only a single RF chain and 1024 passive elements of metasurfaces. These results highlight the potential of wave-domain neural computing to replace digital baseband modules in future wireless transceivers.