We show that if all the finite coset spaces of a polycyclic group have diameter bounded uniformly below by a polynomial in their size then the group is virtually nilpotent. We obtain the same conclusion for a finitely generated residually torsion-free nilpotent group under the weaker assumption that the finite quotient groups have diameter bounded uniformly below by a polynomial in their size. This extends work of Khukhro and Valette.