Strong inter-dependence in complex systems can manifest as partially bipartite networks characterized by interactions occurring primarily between distinct groups of nodes (identified as modules). In this paper, we show that the anti-modular character of such networks, e.g., those defined by the adjacent occurrence of alphabetic characters in corpora of natural language texts, can result in striking structural properties which place them outside the well-known regular/small-world/random network paradigm. Using an ensemble of model networks whose modularity can be tuned, we demonstrate that strong module size heterogeneity in anti-modular random networks imparts them with higher communication efficiency and lower clustering than their randomized counterparts, making them infra small-world. Passage to anti-modularity is associated with characteristic changes in spectral properties of the network, including a delocalization transition exhibited by the principal eigenvector (PEV) of the normalized Laplacian. This is accompanied by the emergence of prominent bimodality in the distribution of PEV components, which can function as a signature for identifying anti-modular organization in empirical networks.