Contact matrices are an important ingredient in age-structured epidemic models to inform the simulated spread of the disease between sub-groups of the population. These matrices are generally derived using resource-intensive diary-based surveys and few exist in the Global South or tailored to vulnerable populations. In particular, no contact matrices exist for refugee settlements - locations under-served by epidemic models in general. In this paper we present a novel, mixed-method approach, for deriving contact matrices in populations which combines a lightweight, rapidly deployable, survey with an agent-based model of the population informed by census and behavioural data. We use this method to derive the first set of contact matrices for the Cox's Bazar refugee settlement in Bangladesh. The matrices from the refugee settlement show strong banding effects due to different age cut-offs in attendance at certain venues, such as distribution centres and religious sites, as well as the important contribution of the demographic profile of the settlement which was encoded in the model. These can have significant implications to the modelled disease dynamics. To validate our approach, we also apply our method to the population of the UK and compare our derived matrices against well-known contact matrices previously collected using traditional approaches. Overall, our findings demonstrate that our mixed-method approach can address some of the challenges of both the traditional and previously proposed agent-based approaches to deriving contact matrices, and has the potential to be rolled-out in other resource-constrained environments. This work therefore contributes to a broader aim of developing new methods and mechanisms of data collection for modelling disease spread in refugee and IDP settlements and better serving these vulnerable communities.