Reproducibility and realistic datasets are crucial for advancing research. Unfortunately, they are often neglected as valid scientific contributions in many young disciplines, with computer science being no exception. In this article, we show the challenges encountered when reproducing the work of others, collecting realistic data in the wild, and ensuring that our own work is reproducible in turn. The presented findings are based on our study investigating the limits of zero-interaction security (ZIS) -- a novel concept, leveraging sensor data collected by Internet of Things (IoT) devices to pair or authenticate devices. In particular, we share our experiences in reproducing five state-of-the-art ZIS schemes, collecting a comprehensive dataset of sensor data from the real world, evaluating these schemes on the collected data, and releasing the data, code, and documentation to facilitate reproducibility of our results. In our discussion, we outline general considerations when conducting similar studies and give specific examples of technical and methodological issues that we experienced. We hope that our findings will raise awareness about the importance of reproducibility and realistic datasets in computer science and inform future research.