We present a new approach to graph limit theory which unifies and generalizes the two most well developed directions, namely dense graph limits (even the more general $L^p$ limits) and Benjamini--Schramm limits (even in the stronger local-global setting). We illustrate by examples that this new framework provides a rich limit theory with natural limit objects for graphs of intermediate density. Moreover, it provides a limit theory for bounded operators (called $P$-operators) of the form $L^\infty(Ω)\to L^1(Ω)$ for probability spaces $Ω$. We introduce a metric to compare $P$-operators (for example finite matrices) even if they act on different spaces. We prove a compactness result which implies that in appropriate norms, limits of uniformly bounded $P$-operators can again be represented by $P$-operators. We show that limits of operators representing graphs are self-adjoint, positivity-preserving $P$-operators called graphops. Graphons, $L^p$ graphons and graphings (known from graph limit theory) are special examples for graphops. We describe a new point of view on random matrix theory using our operator limit framework.