The smallest eigenvalues of (distance-j) Hamming graphs with distance parameter j at least half the length were completely determined by Brouwer et al. (2018). In the present work, we address the complementary regime, namely distances j strictly less than half the length, and derive asymptotic lower bounds on the smallest eigenvalue of binary Hamming graphs. For certain natural generalizations, specifically Cayley graphs defined over quaternary vector spaces, we asymptotically determine the smallest eigenvalue as well. As an application, we obtain lower bounds on the quantum chromatic number of these graphs. In particular, for the aforementioned Cayley graphs over quaternary vectors, our lower bounds for the quantum chromatic number coincide with known upper bounds.