Social fragmentation caused by widening differences among constituents has recently become a highly relevant issue to our modern society. Theoretical models of social fragmentation using the adaptive network framework have been proposed and studied in earlier literature, which are known to either converge to a homogeneous, well-connected network or fragment into many disconnected sub-networks with distinct states. Here we introduced the diversities of behavioral attributes among social constituents and studied their effects on social network evolution. We investigated, using a networked agent-based simulation model, how the resulting network states and topologies would be affected when individual constituents' cultural tolerance, cultural state change rate, and edge weight change rate were systematically diversified. The results showed that the diversity of cultural tolerance had the most direct effect to keep the cultural diversity within the society high and simultaneously reduce the average shortest path length of the social network, which was not previously reported in the earlier literature. Diversities of other behavioral attributes also had effects on final states of the social network, with some nonlinear interactions. Our results suggest that having a broad distribution of cultural tolerance levels within society can help promote the coexistence of cultural diversity and structural connectivity.