The $q$-Whittaker function $W_λ(\mathbf{x};q)$ associated to a partition $λ$ is a $q$-analogue of the Schur function $s_λ(\mathbf{x})$, and is defined as the $t=0$ specialization of the Macdonald polynomial $P_λ(\mathbf{x};q,t)$. We show combinatorially how to expand $W_λ(\mathbf{x};q)$ in terms of partial flags compatible with a nilpotent endomorphism over the finite field of size $1/q$. This yields an expression analogous to a well-known formula for the Hall-Littlewood functions. We show that considering pairs of partial flags and taking Jordan forms leads to a probabilistic bijection between nonnegative-integer matrices and pairs of semistandard tableaux of the same shape, proving the Cauchy identity for $q$-Whittaker functions. We call our probabilistic bijection the $q$-Burge correspondence, and prove that in the limit as $q\to 0$, we recover a description of the classical Burge correspondence (also known as column RSK) due to Rosso (2012). A key step in the proof is the enumeration of an arbitrary double coset of $\text{GL}_n$ modulo two parabolic subgroups, which we find to be of independent interest. As an application, we use the $q$-Burge correspondence to count isomorphism classes of certain modules over the preprojective algebra of a type $A$ quiver (i.e. a path), refined according to their socle filtrations. This develops a connection between the combinatorics of symmetric functions and the representation theory of preprojective algebras.