For $k \ge 2$ and a positive integer $d_0$, we show that if there exists no quaternary Hermitian linear complementary dual $[n,k,d]$ code with $d \ge d_0$ and Hermitian dual distance greater than or equal to $2$, then there exists no quaternary Hermitian linear complementary dual $[n,k,d]$ code with $d \ge d_0$ and Hermitian dual distance $1$. As a consequence, we generalize a result by Araya, Harada and Saito on the nonexistence of some quaternary Hermitian linear complementary dual codes.