Given any positive integer $r$, Nahm's problem is to determine all $r\times r$ rational positive definite matrix $A$, $r$-dimensional rational vector $B$ and rational scalar $C$ such that the rank $r$ Nahm sum associated with $(A,B,C)$ is modular. Around 2007, Zagier conjectured that if the rank $r$ Nahm sum for $(A,B,C)$ is modular, then so is the dual Nahm sum associated with $(A^{-1},A^{-1}B,B^\mathrm{T} A^{-1}B/2-{r}/{24}-C)$. We construct some explicit rank four Nahm sums which are modular while their duals are not modular. This provides counterexamples to Zagier's duality conjecture.