Event-based services have recently witnessed a rapid growth driving the way people explore and share information of interest. They host a huge amount of users' activities including explicit RSVP, shared photos, comments and social connections. Exploiting these activities to detect communities of similar users is a challenging problem. In reality, a community in event-based social network (ESBN) is a group of users not only sharing common events and friends, but also having similar topical interests. However, such community could not be detected by most of existing methods which mainly draw on link analysis in the network. To address this problem, there is a need to capitalize on the semantics of shared objects along with the structural properties, and to generate overlapping communities rather than disjoint ones. In this paper, we propose to leverage the users' activities around events with the aim to detect communities based on topical clustering and link analysis that maximize a new form of semantic modularity. We particularly highlight the difference between online and offline social interactions, and the influence of event categories to detect communities. Experimental results on real datasets showed that our approach was able to detect semantically meaningful communities compared with existing state of the art methods.