We prove that any Cayley graph $G$ with degree $d$ polynomial growth does not satisfy $\{f(n)\}$-containment for any $f=o(n^{d-2})$. This settles the asymptotic behaviour of the firefighter problem on such graphs as it was known that $Cn^{d-2}$ firefighters are enough, answering and strengthening a conjecture of Develin and Hartke. We also prove that intermediate growth Cayley graphs do not satisfy polynomial containment, and give explicit lower bounds depending on the growth rate of the group. These bounds can be further improved when more geometric information is available, such as for Grigorchuk's group.