A general (convex) polytope $P\subset\mathbb R^d$ and its edge-graph $G_P$ can have very distinct symmetry properties. We construct a coloring (of the vertices and edges) of the edge-graph so that the combinatorial symmetry group of the colored edge-graph is isomorphic (in a natural way) to $\mathrm{Aut}_{\mathrm{GL}}(P)$, the group of linear symmetries of the polytope. We also construct an analogous coloring for $\mathrm{Aut}_{\mathrm{O}}(P)$, the group of orthogonal symmetries of $P$.