We present a stochastic method for reconstructing missing spatial and velocity data along the trajectories of small objects passively advected by turbulent flows with a wide range of temporal or spatial scales, such as small balloons in the atmosphere or drifters in the ocean. Our approach makes use of conditional generative diffusion models, a recently proposed data-driven machine learning technique. We solve the problem for two paradigmatic open problems, the case of 3D tracers in homogeneous and isotropic turbulence, and 2D trajectories from the NOAA-funded Global Drifter Program. We show that for both cases, our method is able to reconstruct velocity signals retaining non-trivial scale-by-scale properties that are highly non-Gaussian and intermittent. A key feature of our method is its flexibility in dealing with the location and shape of data gaps, as well as its ability to naturally exploit correlations between different components, leading to superior accuracy, with respect to Gaussian process regressions, for both pointwise reconstruction and statistical expressivity. Our method shows promising applications also to a wide range of other Lagrangian problems, including multi-particle dispersion in turbulence, dynamics of charged particles in astrophysics and plasma physics, and pedestrian dynamics.