Community detection plays an important role in understanding and exploiting the structure of complex systems. Many algorithms have been developed for community detection using modularity maximization or other techniques. In this paper, we formulate the community detection problem as a constrained nonsmooth optimization problem on the compact Stiefel manifold. A Riemannian projected proximal gradient method is proposed and used to solve the problem. To the best of our knowledge, this is the first attempt to use Riemannian optimization for community detection problem. Numerical experimental results on synthetic benchmarks and real-world networks show that our algorithm is effective and outperforms several state-of-art algorithms.