We consider extensions of Brooks' classic theorem on vertex coloring where some colors cannot be used on certain vertices. In particular we prove that if $G$ is a connected graph with maximum degree $Δ(G) \geq 4$ that is not a complete graph and $P \subseteq V(G)$ is a set of vertices where either (i) at most $Δ(G)-2$ colors are forbidden for every vertex in $P$, and any two vertices of $P$ are at distance at least $4$, or (ii) at most $Δ(G)-3$ colors are forbidden for every vertex in $P$, and any two vertices of $P$ are at distance at least $3$, then there is a proper $Δ(G)$-coloring of $G$ respecting these constraints. In fact, we shall prove that these results hold in the more general setting of list colorings. These results are sharp.