Smart speakers are gaining popularity. However, such devices can put the user's privacy at risk whenever hot-words are misinterpreted and voice data is recorded without the user's consent. To mitigate such risks, smart speakers provide privacy control mechanisms like the build-in mute button. Unfortunately, previous work indicated that such mute buttons are rarely used. In this paper, we present the Privacy Hat, a tangible device which can be placed on the smart speaker to prevent the device from listening. We designed the Privacy Hat based on the results of a focus group and developed a working prototype. We hypothesize that the specific user experience of this physical and tangible token makes the use of privacy-enhancing technology more graspable for the user. As a consequence, we expect that the Privacy Hat nudges users to more actively use privacy-enhancing features like the mute button. In addition, we propose the Privacy Hat as a study tool as we hypothesize that the artifact supports participants in reflecting their behaviour. We report on the concept, the prototype and our preliminary results.