Let $N_k(m,n)$ denote the number of partitions of $n$ with Garvan $k$-rank $m$. It is well-known that Andrews-Garvan-Dyson's crank and Dyson's rank are the $k$-rank for $k=1$ and $k=2$, respectively. In this paper, we prove that the sequence $\{N_k(m,n)\}_{|m|\le n-k-71}$ is log-concave for all sufficiently large $n$ and each integer $k$. In particular, we partially solve the log-concavity conjecture for Andrews-Garvan-Dyson's crank and Dyson's rank, which was independently proposed by Bringmann-Jennings-Shaffer-Mahlburg and Ji-Zang.