A natural problem in combinatorial rigidity theory concerns the determination of the rigidity or flexibility of bar-joint frameworks in $\mathbb{R}^d$ that admit some non-trivial symmetry. When $d=2$ there is a large literature on this topic. In particular, it is typical to quotient the symmetric graph by the group and analyse the rigidity of symmetric, but otherwise generic frameworks, using the combinatorial structure of the appropriate group-labelled quotient graph. However, mirroring the situation for generic rigidity, little is known combinatorially when $d\geq 3$. Nevertheless in the periodic case, a key result of Borcea and Streinu characterises when a quotient graph can be lifted to a rigid periodic framework in $\mathbb{R}^d$. We develop an analogous theory for symmetric frameworks in $\mathbb{R}^d$. The results obtained apply to all finite and infinite 2-dimensional point groups, and then in arbitrary dimension they concern a wide range of infinite point groups, sufficiently large finite groups and groups containing translations and rotations. For the case of finite groups we also derive results concerning the probability of assigning group labels to a quotient graph so that the resulting lift is symmetrically rigid in $\mathbb{R}^d$.