Motivated by the $Z$-polynomials of matroids, Ferroni, Matherne, Stevens, and Vecchi introduced the inverse $Z$-polynomial of a matroid. In this paper, we prove several fundamental properties of the inverse $Z$-polynomial, including non-negativity and multiplicativity, and show that it is a valuative invariant. We also provide explicit formulas for the inverse $Z$-polynomials of uniform matroids and a broader class of matroids, namely sparse paving matroids, which include uniform matroids as a special case. Furthermore, we establish the unimodality and log-concavity of these polynomials in the case of sparse paving matroids. Based on the properties of the $Z$-polynomial, we conjecture that the coefficients of the inverse $Z$-polynomial are unimodal and log-concave.