Data collection and aggregation by online services happens to an extent that is often beyond awareness and comprehension of its users. Transparency tools become crucial to inform people, though it is unclear how well they work. To investigate this matter, we conducted a user study focusing on Facebook, which has recently released the "Off-Facebook Activity" transparency dashboard that informs about personal data collection from third parties. We exposed a group of n = 100 participants to the dashboard and surveyed their level of awareness and reactions to understand how transparency impacts users' privacy attitudes and intended behavior. Our participants were surprised about the massive amount of collected data, became significantly less comfortable with data collection, and more likely to take protective measures. Collaterally, we observed that current consent schemes are inadequate. Based on the survey findings, we make recommendations for more usable transparency and highlight the need to raise awareness about transparency tools and to provide easily actionable privacy controls.