We study point configurations on the torus $\mathbb T^d$ that minimize interaction energies with tensor product structure which arise naturally in the context of discrepancy theory and quasi-Monte Carlo integration. Permutation sets on $\mathbb T^2$ and Latin hypercube sets in higher dimensions (i.e. sets whose projections onto coordinate axes are equispaced points) are natural candidates to be energy minimizers. We show that such point configurations that have only one distance in the vector sense minimize the energy for a wide range of potentials, in other words, such sets satisfy a tensor product version of universal optimality. This applies, in particular, to three- and five-point Fibonacci lattices. We also characterize all lattices with this property and exhibit some non-lattice sets of this type. In addition, we obtain several further structural results about global and local minimizers of tensor product energies.