Engaging in interdisciplinary projects on the intersection between visualization and humanities research can be a challenging endeavor. Challenges can be finding valuable outcomes for both domains, or how to apply state-of-the-art visual analytics methods like supervised machine learning algorithms. We discuss these challenges when working with cultural heritage data. Further, there is a gap in applying these methods to intangible heritage. To give a reflection on some interdisciplinary projects, we present three case studies focusing on the labeling of cultural heritage collections, the problems and challenges with the data, the participatory design process, and takeaways for the visualization scholars from these collaborations.