An integer matrix $\mathbf{A}$ is $Δ$-modular if the determinant of each $\text{rank}(\mathbf{A}) \times \text{rank}(\mathbf{A})$ submatrix of $\mathbf{A}$ has absolute value at most $Δ$. The study of $Δ$-modular matrices appears in the theory of integer programming, where an open conjecture is whether integer programs defined by $Δ$-modular constraint matrices can be solved in polynomial time if $Δ$ is considered constant. The conjecture is only known to hold true when $Δ\in \{1,2\}$. In light of this conjecture, a natural question is to understand structural properties of $Δ$-modular matrices. We consider the column number question -- how many nonzero, pairwise non-parallel columns can a rank-$r$ $Δ$-modular matrix have? We prove that for each positive integer $Δ$ and sufficiently large integer $r$, every rank-$r$ $Δ$-modular matrix has at most $\binom{r+1}{2} + 80Δ^7 \cdot r$ nonzero, pairwise non-parallel columns, which is tight up to the term $80Δ^7$. This is the first upper bound of the form $\binom{r+1}{2} + f(Δ)\cdot r$ with $f$ a polynomial function. Underlying our results is a partial list of matrices that cannot exist in a $Δ$-modular matrix. We believe this partial list may be of independent interest in future studies of $Δ$-modular matrices.