We study existence and uniqueness of solutions to the equation $dX_t=b(X_t)dt + dB_t$, where $b$ is a distribution in some Besov space and $B$ is a fractional Brownian motion with Hurst parameter $H\leqslant 1/2$. First, the equation is understood as a nonlinear Young equation. This involves a nonlinear Young integral constructed in the space of functions with finite $p$-variation, which is well suited when $b$ is a measure. Depending on $H$, a condition on the Besov regularity of $b$ is given so that solutions to the equation exist. The construction is deterministic, and $B$ can be replaced by a deterministic path $w$ with a sufficiently smooth local time. Using this construction we prove the existence of weak solutions (in the probabilistic sense). We also prove that solutions coincide with limits of strong solutions obtained by regularisation of $b$. This is used to establish pathwise uniqueness and existence of a strong solution. In particular when $b$ is a finite measure, weak solutions exist for $H<\sqrt{2}-1$, while pathwise uniqueness and strong existence hold when $H\leqslant 1/4$. The proofs involve fine properties of the local time of the fractional Brownian motion, as well as new regularising properties of this process which are established using the stochastic sewing Lemma.