Tensor network states are used extensively as a mathematically convenient description of physically relevant states of many-body quantum systems. Those built on regular lattices, i.e. matrix product states (MPS) in dimension 1 and projected entangled pair states (PEPS) in dimension 2 or higher, are of particular interest in condensed matter physics. The general goal of this work is to characterize which features of MPS and PEPS are generic and which are, on the contrary, exceptional. This problem can be rephrased as follows: given an MPS or PEPS sampled at random, what are the features that it displays with either high or low probability? One property which we are particularly interested in is that of having either rapidly decaying or long-range correlations. In a nutshell, our main result is that translation-invariant MPS and PEPS typically exhibit exponential decay of correlations at a high rate. We have two distinct ways of getting to this conclusion, depending on the dimensional regime under consideration. Both yield intermediate results which are of independent interest, namely: the parent Hamiltonian and the transfer operator of such MPS and PEPS typically have a large spectral gap. In all these statements, our aim is to get a quantitative estimate of the considered quantity (generic correlation length or spectral gap), which has the best possible dependency on the physical and bond dimensions of the random MPS or PEPS.