We study existence of percolation in the hierarchical group of order $N$, which is an ultrametric space, and transience and recurrence of random walks on the percolation clusters. The connection probability on the hierarchical group for two points separated by distance $k$ is of the form $c_k/N^{k(1+δ)}, δ>-1$, with $c_k=C_0+C_1\log k+C_2k^α$, non-negative constants $C_0, C_1, C_2$, and $α>0$. Percolation was proved in Dawson and Gorostiza (2013) for $δ<1$, and for the critical case, $δ=1,C_2>0$, with $α>2$. In this paper we improve the result for the critical case by showing percolation for $α>0$. We use a renormalization method of the type in the previous paper in a new way which is more intrinsic to the model. The proof involves ultrametric random graphs (described in the Introduction). The results for simple (nearest neighbour) random walks on the percolation clusters are: in the case $δ<1$ the walk is transient, and in the critical case $δ=1, C_2>0,α>0$, there exists a critical $α_c\in(0,\infty)$ such that the walk is recurrent for $α<α_c$ and transient for $α>α_c$. The proofs involve graph diameters, path lengths, and electric circuit theory. Some comparisons are made with behaviours of random walks on long-range percolation clusters in the one-dimensional Euclidean lattice.