Crosstalk is defined as the set of unwanted interactions among the different entities of a network. Crosstalk is present in various degrees in every system where information is transmitted through a means that is accessible by all the individual units of the network. Using concepts from graph theory, we introduce a quantifiable measure for sensitivity to crosstalk, and analytically derive the structure of the networks in which it is minimized. It is shown that networks with an inhomogeneous degree distribution are more robust to crosstalk than corresponding homogeneous networks. We provide a method to construct the graph with the minimum possible sensitivity to crosstalk, given its order and size. Finally, for networks with a fixed degree sequence, we present an algorithm to find the optimal interconnection structure among their vertices.