We give simple arithmetic conditions that force the Sylow $p$-subgroup of the critical group of a strongly regular graph to take a specific form. These conditions depend only on the parameters $(v, k, λ, μ)$ of the strongly regular graph under consideration. We give many examples, including how the theory can be used to compute the critical group of Conway's $99$-graph and to give an elementary argument that no $srg(28,9,0,4)$ exists.