We revisit the problem of an elastic line (e.g. a vortex line in a superconductor) subject to both columnar disorder and point disorder in dimension $d=1+1$. Upon applying a transverse field, a delocalization transition is expected, beyond which the line is tilted macroscopically. We investigate this transition in the fixed tilt angle ensemble and within a one-way model where backward jumps are neglected. From recent results about directed polymers and their connections to random matrix theory, we find that for a single line and a single strong defect this transition in presence of point disorder coincides with the Baik-Ben Arous-Peche (BBP) transition for the appearance of outliers in the spectrum of a perturbed random matrix in the GUE. This transition is conveniently described in the polymer picture by a variational calculation. In the delocalized phase, the ground state energy exhibits Tracy-Widom fluctuations. In the localized phase we show, using the variational calculation, that the fluctuations of the occupation length along the columnar defect are described by $f_{KPZ}$, a distribution which appears ubiquitously in the Kardar-Parisi-Zhang universality class. We then consider a smooth density of columnar defect energies. Depending on how this density vanishes at its lower edge we find either (i) a delocalized phase only (ii) a localized phase with a delocalization transition. We analyze this transition which is an infinite-rank extension of the BBP transition. The fluctuations of the ground state energy of a single elastic line in the localized phase (for fixed columnar defect energies) are described by a Fredholm determinant based on a new kernel. The case of many columns and many non-intersecting lines, relevant for the study of the Bose glass phase, is also analyzed. The ground state energy is obtained using free probability and the Burgers equation.