For integers $0 \leq \ell \leq k_{r} \leq k_{c} \leq n$, we give a description for the Smith group of the incidence matrix with rows (columns) indexed by the size $k_r$ ($k_c$, respectively) subsets of an $n$-element set, where incidence means intersection in a set of size $\ell$. This generalizes work of Wilson and Bier from the 1990s which dealt only with the case where incidence meant inclusion. Our approach also describes the Smith group of any matrix in the $\mathbb{Z}$-linear span of these matrices so includes all integer matrices in the Bose-Mesner algebra of the Johnson association scheme: for example, the association matrices themselves as well as the Laplacian, signless Laplacian, Seidel adjacency matrix, etc. of the associated graphs. In particular, we describe the critical (also known as sandpile) groups of these graphs. The complexity of our formula grows with the parameters $k_{r}$ and $k_{c}$, but is independent of $n$ and $\ell$, which often leads to an efficient algorithm for computing these groups. We illustrate our techniques to give diagonal forms of matrices attached to the Kneser and Johnson graphs for subsets of size $3$, whose invariants have never before been described, and recover results from a variety of papers in the literature in a unified way.