We discuss the Grassmann graph $J_q(N,D)$ with $N \geq 2D$, having as vertices the $D$-dimensional subspaces of an $N$-dimensional vector space over the finite field $\mathbb{F}_q$. This graph is distance-regular with diameter $D$; to avoid trivialities we assume $D\geq 3$. Fix a pair of a Delsarte clique $C$ of $J_q(N,D)$ and a vertex $x$ in $C$. We construct a $2D$-dimensional irreducible module $\mathbf{W}$ for the Terwilliger algebra $\mathbf{T}$ of $J_q(N,D)$ associated with the pair $x$, $C$. We show that $\mathbf{W}$ is an irreducible module for the confluent Cherednik algebra $\mathcal{H}_\mathrm{V}$ and describe how the $\mathbf{T}$-action on $\mathbf{W}$ is related to the $\mathcal{H}_\mathrm{V}$-action on $\mathbf{W}$. Using the $\mathcal{H}_\mathrm{V}$-module $\mathbf{W}$, we define non-symmetric dual $q$-Hahn polynomials and prove their recurrence and orthogonality relations from a combinatorial viewpoint.