We enumerate the numbers $Av_n^k(1324)$ of 1324-avoiding $n$-permutations with exactly $k$ inversions for all $k$ and $n \geq (k+7)/2$. The result depends on a structural characterization of such permutations in terms of a new notion of almost-decomposability. In particular, our enumeration verifies half of a conjecture of Claesson, Jelínek and Steingrímsson, according to which $Av_n^k(1324) \leq Av_{n+1}^k(1324)$ for all $n$ and $k$. Proving also the other half would improve the best known upper bound for the exponential growth rate of the number of $1324$-avoiders from $13.5$ to approximately $13.002$.