Many empirical studies suggest that samples of continuous-time signals taken at locations randomly deviated from an equispaced grid (i.e., off-the-grid) can benefit signal acquisition, e.g., undersampling and anti-aliasing. However, explicit statements of such advantages and their respective conditions are scarce in the literature. This paper provides some insight on this topic when the sampling positions are known, with grid deviations generated i.i.d. from a variety of distributions. By solving a square-root LASSO decoder with an interpolation kernel we demonstrate the capabilities of nonuniform samples for compressive sampling, an effective paradigm for undersampling and anti-aliasing. For functions in the Wiener algebra that admit a discrete $s$-sparse representation in some transform domain, we show that $\mathcal{O}(s\log N)$ random off-the-grid samples are sufficient to recover an accurate $\frac{N}{2}$-bandlimited approximation of the signal. For sparse signals (i.e., $s \ll N$), this sampling complexity is a great reduction in comparison to equispaced sampling where $\mathcal{O}(N)$ measurements are needed for the same quality of reconstruction (Nyquist-Shannon sampling theorem). We further consider noise attenuation via oversampling (relative to a desired bandwidth), a standard technique with limited theoretical understanding when the sampling positions are non-equispaced. By solving a least squares problem, we show that $\mathcal{O}(N\log N)$ i.i.d. randomly deviated samples provide an accurate $\frac{N}{2}$-bandlimited approximation of the signal with suppression of the noise energy by a factor $\sim\frac{1}{\sqrt{\log N}}$.