The hole probability, i.e., the probability that a region is void of particles, is a benchmark of correlations in many body systems. We compute analytically this probability $P(R)$ for a spherical region of radius $R$ in the case of $N$ noninteracting fermions in their ground state in a $d$-dimensional trapping potential. Using a connection to the Laguerre-Wishart ensembles of random matrices, we show that, for large $N$ and in the bulk of the Fermi gas, $P(R)$ is described by a universal scaling function of $k_F R$, for which we obtain an exact formula ($k_F$ being the local Fermi wave-vector). It exhibits a super exponential tail $P(R)\propto e^{- κ_d (k_F R)^{d+1}}$ where $κ_d$ is a universal amplitude, in good agreement with existing numerical simulations. When $R$ is of the order of the radius of the Fermi gas, the hole probability is described by a large deviation form which is not universal and which we compute exactly for the harmonic potential. Similar results also hold in momentum space.