For a field $R$ of characteristic $p\ge 0$ and a matrix $c$ in the full $n\times n$ matrix algebra $M_n(R)$ over $R$, let $S_n(c,R)$ be the centralizer algebra of $c$ in $M_n(R)$. We show that $S_n(c,R)$ is a Frobenius-finite, $1$-Auslander-Gorenstein, and gendo-symmetric algebra, and that the extension $S_n(c,R)\subseteq M_n(R)$ is separable and Frobenius. Further, we study the isomorphism problem of invariant matrix algebras. Let $σ$ be a permutation in the symmetric group $Σ_n$ and $c_σ$ the corresponding permutation matrix in $M_n(R)$. We give sufficient and necessary conditions for the invariant algebra $S_n(c_σ,R)$ to be semisimple. If $R$ is an algebraically closed field, we establish a combinatoric characterization of when two semisimple invariant $R$-algebras are isomorphic in terms of the cycle types of permutations.