Let $S_n$ denote the symmetric group on $n$ letters. The $k$-point fixing graph $\mathcal{F}(n,k)$ is defined to be the graph with vertex set $S_n$ and two vertices $g,h$ of $\mathcal{F}(n,k)$ are joined by an edge, if and only if $gh^{-1}$ fixes exactly $k$ points. Ku, Lau and Wong [Cayley graph on symmetric group generated by elements fixing $k$ points, Linear Algebra Appl. 471 (2015) 405-426] obtained a recursive formula for the eigenvalues of $\mathcal{F}(n,k)$. In this paper, we use objects called excited diagrams defined as certain generalizations of skew shapes and derive an explicit formula for the eigenvalues of Cayley graph $\mathcal{F}(n,k)$. Then we apply this formula and show that the eigenvalues of $\mathcal{F}(n,k)$ are in the interval $[\frac{-|S(n,k)|}{n-k-1}, |S(n,k)|]$, where $S(n,k)$ is the set of elements $σ$ of $S_n$ such that $σ$ fixes exactly $k$ points.