The negatively-charged nitrogen vacancy (NV$^-$) centre in diamond is a remarkable optical quantum sensor for a range of applications including, nanoscale thermometry, magnetometry, single photon generation, quantum computing, and communication. However, to date the performance of these techniques using NV$^-$ centres has been limited by the thermally-induced spectral wandering of NV$^-$ centre photoluminescence due to detrimental photothermal heating. Here we demonstrate that solid-state laser refrigeration can be used to enable rapid (ms) optical temperature control of nitrogen vacancy doped nanodiamond (NV$^-$:ND) quantum sensors in both atmospheric and \textit{in vacuo} conditions. Nanodiamonds are attached to ceramic microcrystals including 10\% ytterbium doped yttrium lithium fluoride (Yb:LiYF$_4$) and sodium yttrium fluoride (Yb:NaYF$_4$) by van der Waals bonding. The fluoride crystals were cooled through the efficient emission of upconverted infrared photons excited by a focused 1020 nm laser beam. Heat transfer to the ceramic microcrystals cooled the adjacent NV$^-$:NDs by 10 and 27 K at atmospheric pressure and $\sim$10$^{-3}$ Torr, respectively. The temperature of the NV$^-$:NDs was measured using both Debye-Waller factor (DWF) thermometry and optically detected magnetic resonance (ODMR), which agree with the temperature of the laser cooled ceramic microcrystal. Stabilization of thermally-induced spectral wandering of the NV$^{-}$ zero-phonon-line (ZPL) is achieved by modulating the 1020 nm laser irradiance. The demonstrated cooling of NV$^-$:NDs using an optically cooled microcrystal opens up new possibilities for rapid feedback-controlled cooling of a wide range of nanoscale quantum materials.