We propose a method to search for signs of causal structure in spatiotemporal data making minimal a priori assumptions about the underlying dynamics. To this end, we generalize the elementary concept of recurrence for a point process in time to recurrent events in space and time. An event is defined to be a recurrence of any previous event if it is closer to it in space than all the intervening events. As such, each sequence of recurrences for a given event is a record breaking process. This definition provides a strictly data driven technique to search for structure. Defining events to be nodes, and linking each event to its recurrences, generates a network of recurrent events. Significant deviations in properties of that network compared to networks arising from random processes allows one to infer attributes of the causal dynamics that generate observable correlations in the patterns. We derive analytically a number of properties for the network of recurrent events composed by a random process. We extend the theory of records to treat not only the variable where records happen, but also time as continuous. In this way, we construct a fully symmetric theory of records leading to a number of new results. Those analytic results are compared to the properties of a network synthesized from earthquakes in Southern California. Significant disparities from the ensemble of acausal networks that can be plausibly attributed to the causal structure of seismicity are: (1) Invariance of network statistics with the time span of the events considered, (2) Appearance of a fundamental length scale for recurrences, independent of the time span of the catalog, which is consistent with observations of the ``rupture length'', (3) Hierarchy in the distances and times of subsequent recurrences.