We consider locally isotropic Gaussian random fields on the $N$-dimensional Euclidean space for fixed $N$. Using the so called Gaussian Orthogonally Invariant matrices first studied by Mallows in 1961 which include the celebrated Gaussian Orthogonal Ensemble (GOE), we establish the Kac--Rice representation of expected number of critical points of non-isotropic Gaussian fields, complementing the isotropic case obtained by Cheng and Schwartzman in 2018. In the limit $N=\infty$, we show that such a representation can be always given by GOE matrices, as conjectured by Auffinger and Zeng in 2020.