A Kakeya set $S \subset (\mathbb{Z}/N\mathbb{Z})^n$ is a set containing a line in each direction. We show that, when $N$ is any square-free integer, the size of the smallest Kakeya set in $(\mathbb{Z}/N\mathbb{Z})^n$ is at least $C_{n,ε} N^{n - ε}$ for any $ε$ -- resolving a special case of a conjecture of Hickman and Wright. Previously, such bounds were only known for the case of prime $N$. We also show that the case of general $N$ can be reduced to lower bounding the $\mathbb{F}_p$ rank of the incidence matrix of points and hyperplanes over $(\mathbb{Z}/p^k\mathbb{Z})^n$.