Analog-to-digtial (A/D) conversion plays a crucial role when it comes to the design of energy-efficient and fast signal processing systems. As its complexity grows exponentially with the number of output bits, significant savings are possible when resorting to a minimum resolution of a single bit. However, then the nonlinear effect which is introduced by the A/D converter results in a pronounced performance loss, in particular for the case when the receiver is operated outside the low signal-to-noise ratio (SNR) regime. By trading the A/D resolution for a moderately faster sampling rate, we show that for time-of-arrival (TOA) estimation under any SNR level it is possible to obtain a low-complexity $1$-bit receive system which features a smaller performance degradation then the classical low SNR hard-limiting loss of $2/π$ ($-1.96$ dB). Key to this result is the employment of a lower bound for the Fisher information matrix which enables us to approximate the estimation performance for coarsely quantized receivers with correlated noise models in a pessimistic way.