The sphere graph of $M_r$, a connect sum of $r$ copies of $S^1\times S^2$ was introduced by Hatcher as an analog of the curve graph of a surface to study the outer automorphism group of a free group $F_r$. Bestvina, Bromberg, and Fujiwara proved that the chromatic number of the curve graph is finite; bounds were subsequently improved by Gaster, Greene, and Vlamis. Motivated by the analogy, we provide upper and lower bounds for the chromatic number of the sphere graph of $M_r$. As a corollary to the prime decomposition of 3-manifolds, this gives bounds on the chromatic number of the sphere graph for any orientable 3-manifold.