In 2007, Andrews studied the odd Durfee symbols and their odd ranks. Let $N^0(m,k,n)$ denote the number of odd Durfee symbols of $n$ with odd rank congruent to $m$ modulo $k$. Motivated by Andrews' work, many authors obtained generating functions of $N^0(m,k,n)$ from which relations between odd ranks are proved. In this paper, we establish a family of congruences for odd ranks modulo powers of $5$.