Research has found social VR to bring various benefits to users with and without disabilities. Given the success of social VR applications that support user-created worlds, it is important to consider how we can empower users in building inclusive virtual worlds by investigating how tools for world building can be built to better support collaborations between users with and without disabilities. As such, this position paper provides a brief discussion of existing research into important factors that should be considered during such collaborations, and possible future research directions.