The exponential growth of popularity of multimedia has led to needs for user-centric adaptive applications that manage multimedia content more effectively. Implicit analysis, which examines users' perceptual experience of multimedia by monitoring physiological or behavioral cues, has potential to satisfy such demands. Particularly, physiological signals categorized into cerebral physiological signals (electroencephalography, functional magnetic resonance imaging, and functional near-infrared spectroscopy) and peripheral physiological signals (heart rate, respiration, skin temperature, etc.) have recently received attention along with notable development of wearable physiological sensors. In this paper, we review existing studies on physiological signal analysis exploring perceptual experience of multimedia. Furthermore, we discuss current trends and challenges.