Graphs depict pairwise relationships between objects within a system. Higher-order interactions (HOIs), which involve more than two objects simultaneously, are common in nature. Such interactions can change the stability of a complex system. Hypergraphs can represent an HOI as an arbitrary subset of vertices. However, they fail to capture the specific roles of the vertices involved, which can be highly asymmetric, particularly in the case of interaction modifications. We introduce pangraphs, a robust and quantitative generalisation of graphs that accurately captures arbitrarily complex higher-order interactions. We demonstrate that several higher-order representations proposed in the literature are specific instances of pangraphs. Additionally, we introduce an incidence multilayer digraph representation of a pangraph, referred to as Levi digraph. We adapt degree and Katz centrality measures to the pangraph framework and show that a consistent generalisation of recursive graph measures cannot be simplified to a Levi digraph of a pangraph. We construct a pangraph for a real-world coffee agroecosystem and compare Katz centrality between its dihypergraph and pangraph representations, both analytically and numerically. The choice of representation significantly affects centrality values and alters vertex ranks. Additionally, we emphasise the use of real-valued incidence matrices to quantify interaction strengths and the roles of vertices within the system.