We study the ferromagnetic random field Ising model (RFIM) on a graph $G=(V,E)$ having maximal degree $Δ$, where the external field at each vertex is an i.i.d. random variable. When the random field distribution is sufficiently anti-concentrated, we prove that with high probability over the quenched randomness of the external field, the Glauber dynamics of this RFIM mixes in polynomial time as a consequence of a Poincaré inequality. This model is relevant to the Griffiths phase where the correlations decay exponentially fast in expectation over the quenched random field, but contraction does not hold point-wise due to the existence of weak fields that lead to low-temperature behavior. Previously, fast mixing of Glauber dynamics under large disorder was only proven on the integer lattice, and for RFIM on general graphs, only a sampling algorithm based on self-avoiding walks was known. Under a further technical condition that the random fields are bounded, we prove a modified log-Sobolev inequality for the Glauber dynamics. When the random field is weaker but still satisfies weak spatial mixing (exponential decay of correlations from boundary to bulk) in expectation, and the graph has at most $α$-stretched exponential growth for some $α<1$, then we prove a weak Poincaré inequality holds, which gives rise to a polynomial time sampling algorithm based on Glauber dynamics with warm start. The latter result was previously proven for the integer lattice, and we extend its scope to graphs with only a volume growth condition without assuming a local geometry.