We prove that the saddle connection graph associated to any half-translation surface is 4-hyperbolic and uniformly quasi-isometric to the regular countably infinite-valent tree. Consequently, the saddle connection graph is not quasi-isometrically rigid. We also characterise its Gromov boundary as the set of straight foliations with no saddle connections. In our arguments, we give a generalisation of the unicorn paths in the arc graph which may be of independent interest.