We show that all invertible $n \times n$ matrices over any finite field $\mathbb{F}_q$ can be generated in a Gray code fashion. More specifically, there exists a listing such that (1) each matrix appears exactly once, and (2) two consecutive matrices differ by adding or subtracting one row from a previous or subsequent row, or by multiplying or diving a row by the generator of the multiplicative group of $\mathbb{F}_q$. This even holds if the addition and subtraction of each row is allowed to some specific rows satisfying a certain mild condition. Moreover, we can prescribe the first and the last matrix if $n\ge 3$, or $n=2$ and $q>2$. In other words, the corresponding flip graph on all invertible $n \times n$ matrices over $\mathbb{F}_q$ is Hamilton connected if it is not a cycle. This solves yet another special case of Lovász conjecture on Hamiltonicity of vertex-transitive graphs.