We introduce the notion of \emph{joint spectrum} of a compact set of matrices $S \subset GL_d(\mathbb{C})$, which is a multi-dimensional generalization of the joint spectral radius. We begin with a thorough study of its properties (under various assumptions: irreducibility, Zariski-density, domination). Several classical properties of the joint spectral radius are shown to hold in this generalized setting and an analogue of the Lagarias-Wang finiteness conjecture is discussed. Then we relate the joint spectrum to matrix valued random processes and study what points of it can be realized as Lyapunov vectors. We also show how the joint spectrum encodes all word metrics on reductive groups. Several examples are worked out in detail.