House allocation is an extremely well-studied problem in the field of fair allocation, where the goal is to assign $n$ houses to $n$ agents while satisfying certain fairness criterion, e.g., envy-freeness. To model social interactions, the Graphical House Allocation framework introduces a social graph $G$, in which each vertex corresponds to an agent, and an edge $(u, v)$ corresponds to the potential of agent $u$ to envy the agent $v$, based on their allocations and valuations. In undirected social graphs, the potential for envy is in both the directions. In the Minimum Envy Graphical House Allocation (ME-GHA) problem, given a set of $n$ agents, $n$ houses, a social graph, and agent's valuation functions, the goal is to find an allocation that minimizes the total envy summed up over all the edges of $G$. Recent work, [Hosseini et al., AAMAS 2023, AAMAS 2024] studied ME-GHA in the regime of polynomial-time algorithms, and designed exact and approximation algorithms, for certain graph classes under identical agent valuations. We initiate the study of \gha with non-identical valuations, a setting that has so far remained unexplored. We investigate the multivariate (parameterized) complexity of \gha by identifying structural restrictions on the social graph and valuation functions that yield tractability. We also design moderately exponential-time algorithms for several graph classes, and a polynomial-time algorithm for {binary valuations that returns an allocation with envy at most one when the social graph has maximum degree at most one.