Motivated by problems from compressed sensing, we determine the threshold behavior of a random $n\times d$ $\pm 1$ matrix $M_{n,d}$ with respect to the property "every $s$ columns are linearly independent". In particular, we show that for every $0<δ<1$ and $s=(1-δ)n$, if $d\leq n^{1+1/2(1-δ)-o(1)}$ then with high probability every $s$ columns of $M_{n,d}$ are linearly independent, and if $d\geq n^{1+1/2(1-δ)+o(1)}$ then with high probability there are some $s$ linearly dependent columns.