Social impact evaluations are emerging as a useful tool to understand, document, and evaluate the societal impacts of generative AI. In this provocation, we begin to think carefully about the types of experts and expertise that are needed to conduct robust social impact evaluations of generative AI. We suggest that doing so will require thoughtfully eliciting and integrating insights from a range of "domain experts" and "experiential experts," and close with five open questions.