Internal Diffusion Limited Aggregation is an interacting particle system that describes the growth of a random cluster governed by the boundary harmonic measure seen from an internal point. Our paper studies IDLA in $\mathbb{Z}^d$ driven by critical branching random walks. We prove that, unlike classical IDLA, this process exhibits a phase transition in the dimension. More precisely, we establish the existence of a spherical shape theorem in dimension $d\geq 3$ and the absence of a spherical shape theorem for $d \leq 2$. Our bounds on the inner and outer worst deviations are of polynomial nature, which we expect to be a feature of this model.