Major social media platforms increasingly adopt community-based fact-checking to address misinformation on their platforms. While previous research has largely focused on its effect on engagement (e.g., reposts, likes), an understanding of how fact-checking affects a user's follower base is missing. In this study, we employ quasi-experimental methods to causally assess whether users lose followers after their posts are corrected via community fact-checks. Based on time-series data on follower counts for N=3516 community fact-checked posts from X, we find that community fact-checks do not lead to meaningful declines in the follower counts of users who post misleading content. This suggests that followers of spreaders of misleading posts tend to remain loyal and do not view community fact-checks as a sufficient reason to disengage. Our findings underscore the need for complementary interventions to more effectively disincentivize the production of misinformation on social media.