We identify the dimension of the centralizer of the symmetric group $\mathfrak{S}_d$ in the partition algebra $\mathcal{A}_d(δ)$ and in the Brauer algebra $\mathcal{B}_d(δ)$ with the number of multidigraphs with $d$ arrows and the number of disjoint union of directed cycles with $d$ arrows, respectively. Using Schur-Weyl duality as a fundamental theory, we conclude that each centralizer is related with the $G$-invariant space $P^d(M_n(\mathbf{k}))^G$ of degree $d$ homogeneous polynomials on $n \times n$ matrices, where $G$ is the orthogonal group and the group of permutation matrices, respectively. Our approach gives a uniform way to show that the dimensions of $P^d(M_n(\mathbf{k}))^G$ are stable for sufficiently large $n$.