We study how to establish $\textit{spectral independence}$, a key concept in sampling, without relying on total influence bounds, by applying an $\textit{approximate inverse}$ of the influence matrix. Our method gives constant upper bounds on spectral independence for two foundational Gibbs distributions known to have unbounded total influences: $\bullet$ The monomer-dimer model on graphs with large girth (including trees). Prior to our work, such results were only known for graphs with constant maximum degrees or infinite regular trees, as shown by Chen, Liu, and Vigoda (STOC '21). $\bullet$ The hardcore model on trees with fugacity $λ< \mathrm{e}^2$. This remarkably surpasses the well-known $λ_r>\mathrm{e}-1$ lower bound for the reconstruction threshold on trees, significantly improving upon the current threshold $λ< 1.3$, established in a prior work by Efthymiou, Hayes, Štefankovič, and Vigoda (RANDOM '23). Consequently, we establish optimal $Ω(n^{-1})$ spectral gaps of the Glauber dynamics for these models on arbitrary trees, regardless of the maximum degree $Δ$.