The \textit{toughness} $t(G)$ of a graph $G$ is a measure of its connectivity that is closely related to Hamiltonicity. Brouwer proved the lower bound $t(G) > \ell / λ- 2$ on the toughness of any connected $\ell$-regular graph, where $ λ$ is the largest nontrivial eigenvalue of the adjacency matrix. He conjectured that this lower bound can be improved to $\ell / λ-1$ and this conjecture is still open. Brouwer also observed that many families of graphs (in particular, those achieving equality in the Hoffman ratio bound for the independence number) have toughness exactly $\ell / λ$. Cioabă and Wong confirmed Brouwer's observation for several families of graphs, including Kneser graphs $K(n,2)$ and their complements, with the exception of the Petersen graph $K(5,2)$. In this paper, we extend these results and determine the toughness of Kneser graphs $K(n,k)$ when $k\in \{3,4\}$ and $n\geq 2k+1$ as well as for $k\geq 5$ and sufficiently large $n$ (in terms of $k$). In all these cases, the toughness is attained by the complement of a maximum independent set and we conjecture that this is the case for any $k\geq 5$ and $n\geq 2k+1$.