A major strategy to prevent the spread of COVID-19 is the limiting of in-person contacts. However, this is impractical or impossible for the many disabled people who do not live in care facilities, but still require caregivers. We seek to determine which interventions can prevent infections among disabled people and their caregivers. We simulate transmission with a model that includes susceptible, exposed, asymptomatic, symptomatically ill, hospitalized, and removed individuals. The networks on which we simulate disease spread incorporate heterogeneity in the risks of different types of interactions, time-dependent lockdown and reopening measures, and contact distributions for four different groups (caregivers, disabled people, essential workers, and the general population). We find the probability of becoming infected is largest for caregivers and second largest for disabled people. Our analysis of network structure illustrates that caregivers have the largest modal eigenvector centrality. We find that two interventions -- contact-limiting by all groups and mask-wearing by disabled people and caregivers -- most reduce the cases among disabled people and caregivers. We also test which group spreads COVID-19 most readily by seeding infections in a subset of each group. We find caregivers are the most potent spreaders of COVID-19, particularly to other caregivers and to disabled people. We test where to use limited vaccine doses most effectively and find (1) vaccinating caregivers better protects disabled people than vaccinating the general population or essential workers and (2) vaccinating caregivers protects disabled people about as much as vaccinating disabled people themselves. Our results highlight the potential effectiveness of mask-wearing, contact-limiting throughout society, and strategic vaccination for limiting the exposure of disabled people and their caregivers to COVID-19.