There is a lack of virtual reality (VR) user studies that have been conducted involving people with vision/hearing impairments. This is due to the difficulty of recruiting participants and the accessibility barriers of VR devices. Based on the authors' experience conducting VR user studies with participants with vision/hearing impairments, this position paper identifies 5 key challenges (1. Recruitment, 2. Language Familiarity, 3. Technology Limitations and Barriers, 4. Access to Audio Cue, and 5. Travelling to the Experiment Location) and proposes strategic approaches to mitigate these challenges. In addition, we also presented three key considerations regarding understanding participants' lived experiences that could help the user study become accessible.