Shoulder surfing has been studied extensively, however, it remains unexplored whether and how it impacts users. Understanding this is important as it determines whether shoulder surfing poses a significant concern and, if so, how best to address it. By surveying smartphone users in the UK, we explore how shoulder surfing impacts a) the privacy perceptions of victim users and b) their interaction with smartphones. We found that the impact of being shoulder surfed is highly individual. It is perceived as unavoidable and frequently occurring, leading to increased time for task completion. Individuals are concerned for their own and other peoples privacy, seeing shoulder surfing as a gateway to more serious threats like identity or device theft. Participants expressed a willingness to alter their behaviour and use software based protective measures to prevent shoulder surfing, yet, this comes with a set of user defined criteria, such as effectiveness, affordability, reliability, and availability. We discuss future work directions for user-centred shoulder surfing mitigation.