The origin of the quiet Sun magnetism is under debate. Investigating the solar cycle variation observationally in more detail can give us clues about how to resolve the controversies. We investigate the solar cycle variation of the most magnetically quiet regions and their surface gravity oscillation ($f$-) mode integrated energy ($E_f$). We use 12 years of HMI data and apply a stringent selection criteria, based on spatial and temporal quietness, to avoid any influence of active regions (ARs). We develop an automated high-throughput pipeline to go through all available magnetogram data and to compute $E_f$ for the selected quiet regions. We observe a clear solar cycle dependence of the magnetic field strength in the most quiet regions containing several supergranular cells. For patch sizes smaller than a supergranular cell, no significant cycle dependence is detected. The $E_f$ at the supergranular scale is not constant over time. During the late ascending phase of Cycle 24 (SC24, 2011-2012), it is roughly constant, but starts diminishing in 2013, as the maximum of SC24 is approached. This trend continues until mid-2017, when hints of strengthening at higher southern latitudes are seen. Slow strengthening continues, stronger at higher latitudes than at the equatorial regions, but $E_f$ never returns back to the values seen in 2011-2012. Also, the strengthening trend continues past the solar minimum, to the years when SC25 is already clearly ascending. Hence the $E_f$ behavior is not in phase with the solar cycle. The anticorrelation of $E_f$ with the solar cycle in gross terms is expected, but the phase shift of several years indicates a connection to the poloidal large-scale magnetic field component rather than the toroidal one. Calibrating AR signals with the QS $E_f$ does not reveal significant enhancement of the $f$-mode prior to AR emergence.