There have been incredible advancements in robotics and spoken dialogue systems (SDSs) over the past few years, yet we still don't find social robots in public spaces like train stations, shopping malls, or hospital waiting rooms. In this paper, we argue that early-stage collaboration between robot designers and SDS researchers is crucial to create social robots that can legitimately be used in real-world environments. We draw from our experiences running experiments with social robots, and the surrounding literature, to highlight recurring issues. Robots need more speakers, more microphones, quieter motors, and quieter fans to enable human-robot spoken interaction in the wild and improve accessibility. More robust robot joints are also needed to limit potential harm to older adults and other more vulnerable groups.