Context: In agile transformations, there are many challenges such as alignment between agile practices and the organizational goals and strategies or issues with shifts in how work is organized and executed. One very important challenge but less considered and treated in research are cultural challenges associated with an agile mindset. Although research shows that cultural clashes and general organizational resistance to change are part of the most significant agile adoption barriers. Objective: We identify challenges that arise from the interplay between agile culture and organizational culture. In doing so, we tackle this field and come up with important contributions for further research regarding a problem that practitioners face today. Method: This is done with a mixed-method research approach. First, we gathered qualitative data among our network of agile practitioners and derived in sum 15 challenges with agile culture. Then, we conducted quantitative data by means of a questionnaire study with 92 participants. Results: We identified 7 key challenges out of the 15 challenges with agile culture. These key challenges refer to the technical agility (doing agile) and the cultural agility (being agile). The results are presented in type of a conceptual model named the Agile Cultural Challenges (ACuCa). Conclusion: Based on our results, we started deriving future work aspects to do more detailed research on the topic of cultural challenges while transitioning or using agile methods in software development and beyond.