It was recently shown by Lubetzky and Peres (2016) and by Sardari (2018) that Ramanujan graphs, i.e., graphs with the optimal spectrum, exhibit cutoff of the simple random walk in an optimal time and have an optimal almost-diameter. We show that this spectral condition can be replaced by a weaker condition, the Sarnak-Xue density property, to deduce similar results. This allows us to prove that some natural families of Schreier graphs of the $SL_2(\mathbb{F}_t)$-action on the projective line exhibit cutoff, thus proving a special case of a conjecture of Rivin and Sardari.