In this paper we show that for integers $s\geq2$, $t\geq1$, any co-edge-regular graph which is cospectral with the $s$-clique extension of the $t\times t$-grid is the $s$-clique extension of the $t\times t$-grid, if $t$ is large enough. Gavrilyuk and Koolen used a weaker version of this result to show that the Grassmann graph $J_q(2D,D)$ is characterized by its intersection array as a distance-regular graph, if $D$ is large enough.