Motivated by the study of polytopes formed as the convex hull of permutation matrices and alternating sign matrices, we define several new families of polytopes as convex hulls of sign matrices, which are certain {0,1,-1}-matrices in bijection with semistandard Young tableaux. We investigate various properties of these polytopes, including their inequality descriptions, vertices, facets, and face lattices, as well as connections to alternating sign matrix polytopes and transportation polytopes.