We propose a generalization of the iterative closest point (ICP) algorithm for point set registration, in which the registration functions are non-rigid and follow the large deformation diffeomorphic metric mapping (LDDMM) framework. The algorithm is formulated as a well-posed probabilistic inference, and requires to solve a novel variation of LDDMM landmark registration with an additional term involving the Jacobian of the mapping. The algorithm can easily be generalized to construct a diffeomorphic, statistical atlas of multiple point sets. The method is successfully validated on a first set of synthetic data.