Mining spatio-temporal correlation patterns for traffic prediction is a well-studied field. However, most approaches are based on the assumption of the availability of and accessibility to a sufficiently dense data source, which is rather the rare case in reality. Traffic sensors in road networks are generally highly sparse in their distribution: fleet-based traffic sensing is sparse in space but also sparse in time. There are also other traffic application, besides road traffic, like moving objects in the marine space, where observations are sparsely and arbitrarily distributed in space. In this paper, we tackle the problem of traffic prediction on sparse and spatially irregular and non-deterministic traffic observations. We draw a border between imputations and this work as we consider high sparsity rates and no fixed sensor locations. We advance correlation mining methods with a Sparse Unstructured Spatio Temporal Reconstruction (SUSTeR) framework that reconstructs traffic states from sparse non-stationary observations. For the prediction the framework creates a hidden context traffic state which is enriched in a residual fashion with each observation. Such an assimilated hidden traffic state can be used by existing traffic prediction methods to predict future traffic states. We query these states with query locations from the spatial domain.