We study the modular representation theory of rank $3$ association schemes arising from partial geometries with parameters $(s,t,α)$. First, we obtain an explicit closed formula for the Frame number of the point scheme in terms of the number of points $v$ and the parameter $s+t+1-α$, and use it to characterize the primes $p$ for which the adjacency algebra over $\mathbb{F}_p$ is not semisimple. We then give a complete case-by-case description of the Jacobson radical of this algebra in four arithmetic situations and determine the generic $p$-ranks of the adjacency matrices. As a step toward understanding the modular representation theory of coherent configurations of type $[3,2;3]$ associated with strongly regular designs, we analyze the relationship between the modular structure of the point scheme and that of the design algebra. For the generalized quadrangle $\mathrm{GQ}(2,2)$ we obtain partial results on the structure of the $2$-modular adjacency algebra $\mathbb{F}_2 \mathfrak{X}$, and we explain the representation-theoretic difficulties that prevent a complete determination of its Wedderburn decomposition and Gabriel quiver, which remains open and is formulated as Problem~6.8.