Social media is nearly ubiquitous in modern life, raising concerns about its societal impacts -- from mental health and polarization to violence and democratic disruption. Yet research on its causal effects is still inconclusive: Various methods, spanning observational to experimental, can yield seemingly conflicting results. Considering the complexity of such socio-technical systems, with coupled networks, feedback loops and collective phenomena, this may not be surprising. Here, we enumerate and examine the features of social media as a complex system that challenge our ability to infer causality at societal scales. Attempts to ascertain and summarize causal effects have tended to prioritize findings from randomized controlled trials (RCTs). However, like observational studies, RCTs rely on assumptions that may frequently be violated in the context of social media, especially regarding societal outcomes at scale. Drawing on insight from disciplines that have faced similar challenges, like climate-science or epidemiology, we propose a path forward that combines the strengths of observational and experimental approaches while acknowledging the limitations of each. Progress, we argue, requires moving beyond isolated, linear effects to mechanistic explanations of how social media platforms generate collective outcomes.