The popularity of voice-controlled smart speakers with intelligent personal assistants (IPAs) like the Amazon Echo and their increasing use as an interface for other Internet of Things (IoT) technologies in the home provides opportunities to study smart speakers as an emerging and ubiquitous IoT device. Prior research has studied how smart speaker usage has unfolded in homes and how the devices have been integrated into people's daily routines. In this context, researchers have also observed instances of smart speakers' 'black box' behaviour. In this position paper, we present findings from a study we conducted to specifically investigate such challenges people experience around intelligibility and control of their smart speakers, for instance, when the smart speaker interfaces with other IoT systems. Reflecting on our findings, we discuss new possible directions for smart speakers including physical intelligibility, situational physical interaction, and providing access to alternative interpretations in shared and non-shared contexts.