The purpose of this paper is to give explicit constructions of unitary $t$-designs in the unitary group $U(d)$ for all $t$ and $d$. It seems that the explicit constructions were so far known only for very special cases. Here explicit construction means that the entries of the unitary matrices are given by the values of elementary functions at the root of some given polynomials. We will discuss what are the best such unitary $4$-designs in $U(4)$ obtained by these methods. Indeed we give an inductive construction of designs on compact groups by using Gelfand pairs $(G,K)$. Note that $(U(n),U(m) \times U(n-m))$ is a Gelfand pair. By using the zonal spherical functions for $(G,K)$, we can construct designs on $G$ from designs on $K$. We remark that our proofs use the representation theory of compact groups crucially. We also remark that this method can be applied to the orthogonal groups $O(d)$, and thus provides another explicit construction of spherical $t$-designs on the $d$ dimensional sphere $S^{d-1}$ by the induction on $d$.