In the dynamic metric $k$-median problem, we wish to maintain a set of $k$ centers $S \subseteq V$ in an input metric space $(V, d)$ that gets updated via point insertions/deletions, so as to minimize the objective $\sum_{x \in V} \min_{y \in S} d(x, y)$. The quality of a dynamic algorithm is measured in terms of its approximation ratio, "recourse" (the number of changes in $S$ per update) and "update time" (the time it takes to handle an update). The ultimate goal in this line of research is to obtain a dynamic $O(1)$ approximation algorithm with $\tilde{O}(1)$ recourse and $\tilde{O}(k)$ update time. Dynamic $k$-median is a canonical example of a class of problems known as dynamic $k$-clustering, that has received significant attention in recent years. To the best of our knowledge, however, previous papers either attempt to minimize the algorithm's recourse while ignoring its update time, or minimize the algorithm's update time while ignoring its recourse. For dynamic $k$-median, we come arbitrarily close to resolving the main open question on this topic, with the following results. (I) We develop a new framework of randomized local search that is suitable for adaptation in a dynamic setting. For every $ε> 0$, this gives us a dynamic $k$-median algorithm with $O(1/ε)$ approximation ratio, $\tilde{O}(k^ε)$ recourse and $\tilde{O}(k^{1+ε})$ update time. This framework also generalizes to dynamic $k$-clustering with $\ell^p$-norm objectives, giving similar bounds for the dynamic $k$-means and a new trade-off for dynamic $k$-center. (II) If it suffices to maintain only an estimate of the value of the optimal $k$-median objective, then we obtain a $O(1)$ approximation algorithm with $\tilde{O}(k)$ update time. We achieve this result via adapting the Lagrangian Relaxation framework to the dynamic setting.