We study a refinement of the $q,t$-Catalan numbers introduced by Xin and Zhang (2022, 2023) using tools from polyhedral geometry. These refined $q,t$-Catalan numbers depend on a vector of parameters $\vec{k}$ and the classical $q,t$-Catalan numbers are recovered when $\vec{k} = (1,\ldots,1)$. We interpret Xin and Zhang's generating functions by developing polyhedral cones arising from constraints on $\vec{k}$-Dyck paths and their associated area and bounce statistics. Through this polyhedral approach, we recover Xin and Zhang's theorem on $q,t$-symmetry of the refined $q,t$-Catalan numbers in the cases where $\vec{k} = (k_1,k_2,k_3)$ and $(k,k,k,k)$, give some extensions, including the case $\vec{k} = (k,k+m,k+m,k+m)$, and discuss relationships to other generalizations of the $q,t$-Catalan numbers.