For small $n$, the known compact hyperbolic $n$-orbifolds of minimal volume are intimately related to Coxeter groups of smallest rank. For $n=2$ and $3$, these Coxeter groups are given by the triangle group $[7,3]$ and the tetrahedral group $[3,5,3]$, and they are also distinguished by the fact that they have minimal growth rate among all cocompact hyperbolic Coxeter groups in $\hbox{Isom}\mathbb H^n$, respectively. In this work, we consider the cocompact Coxeter simplex group $G_4$ with Coxeter symbol $[5,3,3,3]$ in $\hbox{Isom}\mathbb H^4$ and the cocompact Coxeter prism group $G_5$ based on $[5,3,3,3,3]$ in $\hbox{Isom}\mathbb H^5$. Both groups are arithmetic and related to the fundamental group of the minimal volume arithmetic compact hyperbolic $n$-orbifold for $n=4$ and $5$, respectively. Here, we prove that the group $G_n$ is distinguished by having smallest growth rate among all Coxeter groups acting cocompactly on $\mathbb H^n$ for $n=4$ and $5$, respectively. The proof is based on combinatorial properties of compact hyperbolic Coxeter polyhedra, some partial classification results and certain monotonicity properties of growth rates of the associated Coxeter groups.