Character polynomials are used to study the restriction of a polynomial representation of a general linear group to its subgroup of permutation matrices. A simple formula is obtained for computing inner products of class functions given by character polynomials. Character polynomials for symmetric and alternating tensors are computed using generating functions with Eulerian factorizations. These are used to compute character polynomials for Weyl modules, which exhibit a duality. By taking inner products of character polynomials for Weyl modules and character polynomials for Specht modules, stable restriction coefficients are easily computed. Generating functions of dimensions of symmetric group invariants in Weyl modules are obtained. Partitions with two rows, two columns, and hook partitions whose Weyl modules have non-zero vectors invariant under the symmetric group are characterized. A reformulation of the restriction problem in terms of a restriction functor from the category of strict polynomial functors to the category of finitely generated FI-modules is obtained.