Motivated by the question of whether a random polynomial with integer coefficients is likely to be irreducible, we study the probability that a monic polynomial with integer coefficients has a low-degree factor over the integers, which is equivalent to having a low-degree algebraic root. It is known in certain cases that random polynomials with integer coefficients are very likely to be irreducible, and our project can be viewed as part of the general program of testing whether this is a universal behavior exhibited by many random polynomial models. Our main result shows that pointwise delocalization of the roots of a random polynomial can be used to imply that the polynomial is unlikely to have a low-degree factor over the integers. We apply our main result to a number of models of random polynomials, including characteristic polynomials of random matrices, where strong delocalization results are known. Studying a variety of random matrix models---including iid matrices, symmetric matrices, elliptical matrices, and adjacency matrices of random graphs and digraphs---we show that, for a random square matrix with integer entries, the characteristic polynomial is unlikely to have a low-degree factor over the integers, which is equivalent to the matrix having an eigenvalue that is algebraic with low degree. Having a low-degree algebraic eigenvalue generalizes the questions of whether the matrix has a rational eigenvalue and whether the matrix is singular (i.e., has an eigenvalue equal to zero).