The existence of inter-dependence between multiple networks imparts an additional scale of complexity to such systems often referred to as `network of networks' (NON). We have investigated the robustness of NONs to random breakdown of their components, as well as targeted attacks, as a function of the relative proportion of intra- and inter-dependence among the constituent networks. We focus on bi-layer networks with the two layers comprising different number of nodes in general and where the ratio of intra-layer to inter-layer connections, $r$, can be varied, keeping the total number of nodes and overall connection density invariant. We observe that while the responses of the different networks to random breakdown of nodes are similar, dominantly intra-dependent networks ($r\ll1$) are robust with respect to attacks that target nodes having highest degree but when nodes are removed on the basis of highest betweenness centrality (CB), they exhibit a sharp decrease in the size of the largest connected component (resembling a first order phase transition) followed by a more gradual decrease as more nodes are removed (akin to a second order transition). We also explore the role of layer size heterogeneity on robustness, finding that for a given $r$ having layers comprising very different number of nodes results in a bimodal degree distribution. For dominantly inter-dependent networks, this results in the nodes of the smaller layer becoming structurally central. Selective removal of these nodes, which constitute a relatively small fraction of the network, leads to breakdown of the entire system - making the inter-dependent networks even more fragile to targeted attacks than scale-free networks having power-law degree distribution.