We show that the strongly regular graph on non-isotropic points of one type of the polar spaces of type $U(n, 2)$, $O(n, 3)$, $O(n, 5)$, $O^+(n, 3)$, and $O^-(n, 3)$ are not determined by its parameters for $n \geq 6$. We prove this by using a variation of Godsil-McKay switching recently described by Wang, Qiu, and Hu. This also results in a new, shorter proof of a previous result of the first author which showed that the collinearity graph of a polar space is not determined by its spectrum. The same switching gives a linear algebra explanation for the construction of a large number of non-isomorphic designs.