In this paper we investigate the structure of the critical groups of doubly regular tournaments (DRTs) associated with skew Hadamard difference families (SDFs) with one, two, or four blocks. Brown and Reid found the existence of a skew Hadamard matrix of order $n+1$ to be equivalent to the existence of a DRT on $n$ vertices. A well known construction of a skew Hadamard matrix order $n$ is by constructing skew Hadamard difference sets in abelian groups of order $n-1$. The Paley skew Hadamard matrix is an example of one such construction. Szekeres and Whiteman constructed skew Hadamard matrices from skew Hadamard difference families with two blocks. Wallis and Whiteman constructed skew Hadamard matrices from skew Hadamard difference families with four blocks. In this paper we consider the critical groups of DRTs associated with skew Hadamard matrices constructed from skew Hadamard difference families with one, two or four blocks. We compute the critical groups of DRTs associated with skew Hadamard difference families with two or four blocks. We also compute the critical group of the Paley tournament and show that this tournament is inequivalent to the other DRTs we considered. Consequently we prove that the associated skew Hadamard matrices are not equivalent.