Estimating the unitarity of an unknown quantum channel $\mathcal{E}$ provides information on how much it is unitary, which is a basic and important problem in quantum device certification and benchmarking. Unitarity estimation can be performed with either coherent or incoherent access, where the former in general leads to better query complexity while the latter allows more practical implementations. In this paper, we provide a unified framework for unitarity estimation, which induces ancilla-efficient algorithms that use $O(ε^{-2})$ and $O(\sqrt{d}\cdotε^{-2})$ calls to $\mathcal{E}$ with coherent and incoherent accesses, respectively, where $d$ is the dimension of the system that $\mathcal{E}$ acts on and $ε$ is the required precision. We further show that both the $d$-dependence and $ε$-dependence of our algorithms are optimal. As part of our results, we settle the query complexity of the distinguishing problem for depolarizing and unitary channels with incoherent access by giving a matching lower bound $Ω(\sqrt{d})$, improving the prior best lower bound $Ω(\sqrt[3]{d})$ by Aharonov et al. (Nat. Commun. 2022) and Chen et al. (FOCS 2021).