In this paper, we study association schemes on the anisotropic points of classical polar spaces. Our main result concerns non-degenerate elliptic and hyperbolic quadrics in PG$(n,q)$ with $q$ odd. We define relations on the anisotropic points of such a quadric that depend on the type of line spanned by the points and whether or not they are of the same "quadratic type". This yields an imprimitive $5$-class association scheme. We calculate the matrices of eigenvalues and dual eigenvalues of this scheme. We also use this result, together with similar results from the literature concerning other classical polar spaces, to exactly calculate the spectrum of orthogonality graphs on the anisotropic points of non-degenerate quadrics in odd characteristic and of non-degenerate Hermitian varieties. As a byproduct, we obtain a $3$-class association scheme on the anisotropic points of non-degenerate Hermitian varieties, where the relation containing two points depends on the type of line spanned by these points, and whether or not they are orthogonal.