In this paper, we obtain bounds on the Wilson loop expectations in 4D $U(1)$ lattice gauge theory which quantify the effect of topological defects. In the case of a Villain interaction, by extending the non-perturbative technique introduced in [GS20a], we obtain the following estimate for a large loop $γ$ at low temperatures: \[ |\langle W_γ\rangle_β| \leq \exp \left(-\frac{C_{GFF}} {2β}(1+C βe^{- 2π^2 β} )(|γ|+o(|γ|)) \right)\,. \] Our result is in the line of recent works [Cha20, Cao20, FLV20, For21] which analyze the case where the gauge group is discrete. In the present case where the gauge group is continuous and Abelian, the fluctuations of the gauge field decouple into a Gaussian part, related to the so-called {\em free electromagnetic wave} [Gro83, Dri87], and a gas of {\em topological defects}. As such, our work gives new quantitative bounds on the fluctuations of the latter which complement the works by Guth and Fröhlich-Spencer [Gut80, FS82]. Finally, we improve, also in a non-perturbative way, the correction term from $e^{-2π^2β}$ to $e^{-π^2β}$ in the case of the free-energy of the system. This provides a matching lower-bound with the prediction of Guth [Gut80] based on renormalization group techniques.