Parameterized complexity seeks to use input structure to obtain faster algorithms for NP-hard problems. This has been most successful for graphs of low treewidth: Many problems admit fast algorithms relative to treewidth and many of them are optimal under SETH. Fewer such results are known for more general structure such as low clique-width and more restrictive structure such as low deletion distance to a sparse graph class. Despite these successes, such results remain "islands'' within the realm of possible structure. Rather than adding more islands, we seek to determine the transitions between them, that is, we aim for structural thresholds where the complexity increases as input structure becomes more general. Going from deletion distance to treewidth, is a single deletion set to a graph with simple components enough to yield the same lower bound as for treewidth or does it take many disjoint separators? Going from treewidth to clique-width, how much more density entails the same complexity as clique-width? Conversely, what is the most restrictive structure that yields the same lower bound? For treewidth, we obtain both refined and new lower bounds that apply already to graphs with a single separator $X$ such that $G-X$ has treewidth $r=O(1)$, while $G$ has treewidth $|X|+O(1)$. We rule out algorithms running in time $O^*((r+1-ε)^{k})$ for Deletion to $r$-Colorable parameterized by $k=|X|$. For clique-width, we rule out time $O^*((2^r-ε)^k)$ for Deletion to $r$-Colorable, where $X$ is now allowed to consist of $k$ twinclasses. There are further results on Vertex Cover, Dominating Set and Maximum Cut. All lower bounds are matched by existing and newly designed algorithms.