Context: Within agile transformations, there are a lot of different challenges coming up. One very important but less considered and treated in research are cultural challenges. Although research shows that cultural clashes and general organizational resistance to change are part of the most significant agile adoption barriers. Objective: Thus, our objective is to tackle this field and come up with important contributions for further research. To this end, we want to identify challenges that arise from the interplay between agility and organizational culture. Method: This is done based on an iterative research approach. On the one hand, we gathered qualitative data among our network of agile practitioners. Then, we derived in sum 15 challenges with agile culture. On the other hand, we gathered quantitative data by means of a questionnaire study with 92 participants. Results: We identified 7 key challenges out of the 15 challenges with agile culture. The results that are presented in a conceptual model show a focus on human aspects that we need to deal with more in future. Conclusion: Based on our results, we started deriving future work aspects to do more detailed research on the topic of cultural challenges while transitioning or using agile methods in software development and beyond.