We study the fine-grained complexity of counting the number of colorings and connected spanning edge sets parameterized by the cutwidth and treewidth of the graph. While decompositions of small treewidth decompose the graph with small vertex separators, decompositions with small cutwidth decompose the graph with small \emph{edge} separators. Let $p,q \in \mathbb{N}$ such that $p$ is a prime and $q \geq 3$. - If $p$ divides $q-1$, there is a $(q-1)^{\text{ctw}}n^{O(1)}$ time algorithm for counting list $q$-colorings modulo $p$ of $n$-vertex graphs of cutwidth $\text{ctw}$ and for all $\varepsilon>0$ there is no algorithm running in time $(q-1-\varepsilon)^{\text{ctw}} n^{O(1)}$, assuming the Strong Exponential Time Hypothesis (SETH). - If $p$ does not divide $q-1$, there is a (folklore) $q^{\text{ctw}}n^{O(1)}$ time algorithm for counting list $q$-colorings modulo $p$ of $n$-vertex graphs of cutwidth $\text{ctw}$ and for all $\varepsilon>0$ there is no algorithm running in time $(q-\varepsilon)^{\text{ctw}} n^{O(1)}$, assuming SETH. The lower bounds are in stark contrast with the existing $2^{\text{ctw}}n^{O(1)}$ time algorithm to compute the chromatic number of a graph by Jansen and Nederlof~[Theor. Comput. Sci.'18]. Both our algorithms and lower bounds employ use of the matrix rank method, by relating the complexity of the problem to the rank of a certain `compatibility matrix' in a non-trivial way. We extend our lower bounds to counting connected spanning edge sets modulo $p$ and give an algorithm with matching running time for both treewidth and cutwidth.