We consider Kempe changes on the $k$-colorings of a graph on $n$ vertices. If the graph is $(k-1)$-degenerate, then all its $k$-colorings are equivalent up to Kempe changes. However, the sequence between two $k$-colorings that arises from the proof may be exponential in the number of vertices. An intriguing open question is whether it can be turned polynomial. We prove this to be possible under the stronger assumption that the graph has treewidth at most $k-1$. Namely, any two $k$-colorings are equivalent up to $O(kn^2)$ Kempe changes. We investigate other restrictions (list coloring, bounded maximum average degree, degree bounds). As a main result, we derive that given an $n$-vertex graph with maximum degree $Δ$, the $Δ$-colorings are all equivalent up to $O(n^2)$ Kempe changes, unless $Δ= 3$ and some connected component is a 3-prism.