Graph topology identification (GTI) is a central challenge in networked systems, where the underlying structure is often hidden, yet nodal data are available. Conventional solutions to address these challenges rely on probabilistic models or complex optimization formulations, commonly suffering from non-convexity or requiring restrictive assumptions on acyclicity or positivity. In this paper, we propose a novel covariance matching (CovMatch) framework that directly aligns the empirical covariance of the observed data with the theoretical covariance implied by an underlying graph. We show that as long as the data-generating process permits an explicit covariance expression, CovMatch offers a unified route to topology inference. We showcase our methodology on linear structural equation models (SEMs), showing that CovMatch naturally handles both undirected and general sparse directed graphs - whether acyclic or positively weighted - without explicit knowledge of these structural constraints. Through appropriate reparameterizations, CovMatch simplifies the graph learning problem to either a conic mixed integer program for undirected graphs or an orthogonal matrix optimization for directed graphs. Numerical results confirm that, even for relatively large graphs, our approach efficiently recovers the true topology and outperforms standard baselines in accuracy. These findings highlight CovMatch as a powerful alternative to log-determinant or Bayesian methods for GTI, paving the way for broader research on learning complex network topologies with minimal assumptions.