We consider the stochastic differential equation $$ X_t = x_0 + \int_0^t f(X_s)ds + \int_0^tσ(X_s)dB^{H}_s,$$ with $x_0 \in \mathbb{R}^d$, $d \geq 1$, $f: \mathbb{R}^d \rightarrow \mathbb{R}^d$ is bounded continuous, $σ: \mathbb{R}^d \rightarrow \mathbb{R}^{d\times d}$ is a uniformly elliptic, bounded, twice continuously differentiable conservative vector field and $B^H$ is fractional Brownian motion with $H \in (\frac{1}{3}, \frac{1}{2}]$. When $d=1$, $H= \frac{1}{2}$, and $f$ is Hölder continuous, in the spirit of Davie [D07], we establish the existence of a null set $\mathcal{N}$ depending only on $f, σ$ such that for all $x_0\in \mathbb{R}$ and $ω\in Ω\setminus \mathcal{N}$, the above equation admits a path-by-path unique solution. Our proof is based on establishing the uniform continuous differentiability of the flow associated with the equation. We also establish the path-by-path uniqueness for $d \geq 1$ and $H \in (\frac{1}{3}, \frac{1}{2}]$, but the null set may depend on $x_0$, thus extending a result of Catellier-Gubinelli [CG12].