The Terwilliger algebra $T(x)$ of a finite connected simple graph $Γ$ with respect to a vertex $x$ is the complex semisimple matrix algebra generated by the adjacency matrix $A$ of $Γ$ and the diagonal matrices $E_i^*(x)=\operatorname{diag}(v_i)$ $(i=0,1,2,\dots)$, where $v_i$ denotes the characteristic vector of the set of vertices at distance $i$ from $x$. The twisted Grassmann graph $\tilde{J}_q(2D+1,D)$ discovered by Van Dam and Koolen in 2005 has two orbits of the automorphism group on its vertex set, and it is known that one of the orbits has the property that $T(x)$ is thin whenever $x$ is chosen from it, i.e., every irreducible $T(x)$-module $W$ satisfies $\dim E_i^*(x)W\leqslant 1$ for all $i$. In this paper, we determine all the irreducible $T(x)$-modules of $\tilde{J}_q(2D+1,D)$ for this "thin" case.