The expected Euler characteristic (EEC) method is an integral-geometric method used to approximate the tail probability of the maximum of a random field on a manifold. Noting that the largest eigenvalue of a real-symmetric or Hermitian matrix is the maximum of the quadratic form of a unit vector, we provide EEC approximation formulas for the tail probability of the largest eigenvalue of orthogonally invariant random matrices of a large class. For this purpose, we propose a version of a skew-orthogonal polynomial by adding a side condition such that it is uniquely defined, and describe the EEC formulas in terms of the (skew-)orthogonal polynomials. In addition, for the classical random matrices (Gaussian, Wishart, and multivariate beta matrices), we analyze the limiting behavior of the EEC approximation as the matrix size goes to infinity under the so-called edge-asymptotic normalization. It is shown that the limit of the EEC formula approximates well the Tracy-Widom distributions in the upper tail area, as does the EEC formula when the matrix size is finite.