When a new infectious disease (or a new strain of an existing one) emerges, as in the recent COVID-19 pandemic, different types of mobility restrictions are considered to slow down or mitigate the spread of the disease. The measures to be adopted require carefully weighing the social cost against their impact on disease control. In this work, we analyze, in a context of mobility restrictions, the role of frequent versus occasional contacts in epidemic spread. We develop an individual-based mathematical model where frequent contacts among individuals (at home, work, schools) and occasional contacts (at stores, transport, etc.) are considered. We define several contact structures by varying the relative weight of frequent and occasional contacts while keeping the same initial growth rate of the epidemic. We find the remarkable result that the more frequent contacts prevail over occasional ones, the higher the epidemic peak, the sooner it occurs, and the greater the final number of individuals affected by the epidemic. We conduct our study using an SIR model, considering both exponential and deterministic recovery from infection, and obtain that this effect is more pronounced under deterministic recovery. We find that the impact of relaxation measures depends on the relative importance of frequent and occasional contacts within the considered social structures. Finally, we assess in which of the considered scenarios the homogeneous mixing approximation provides a reasonable description of the epidemic dynamics.