The Ehrhart polynomial $ehr_P (n)$ of a lattice polytope $P$ gives the number of integer lattice points in the $n$-th dilate of $P$ for all integers $n\geq 0$. The degree of $P$ is defined as the degree of its $h^\ast$-polynomial, a particular transformation of the Ehrhart polynomial with many useful properties which serves as an important tool for classification questions in Ehrhart theory. A zonotope is the Minkowski (pointwise) sum of line segments. We classify all Ehrhart polynomials of lattice zonotopes of degree $2$ thereby complementing results of Scott (1976), Treutlein (2010), and Henk-Tagami (2009). Our proof is constructive: by considering solid-angles and the lattice width, we provide a characterization of all $3$-dimensional zonotopes of degree $2$.