This paper develops a framework for establishing the existence of solutions to the equilibrium Hamilton-Jacobi-Bellman (EHJB) equation arising in time-inconsistent stochastic control problems. The time-inconsistency in our setting arises from the initial-time dependence such as the non-exponential discounting. The classical approach typically relates the existence of equilibrium to the classical solution of the EHJB, whose existence is still an open problem under general model assumptions. We resolve this challenge by building on a vanishing entropy regularization approach. Using fixed-point arguments, we first establish the existence of classical solutions to the exploratory equilibrium Hamilton-Jacobi-Bellman Equation (EEHJB) by deriving a series of delicate PDE estimates for the solution and its derivatives. Building on these estimates for the solution of the EEHJB and its derivatives, we then conduct a rigorous convergence analysis under suitable norms as the entropy regularization vanishes. Our main result shows that solutions of the EEHJB converge to a strong solution of the original EHJB, corresponding to the limit of the regularized equilibria. This convergence yields a verification argument ensuring that the limiting relaxed equilibrium indeed constitutes an equilibrium for the original time-inconsistent control problem. We thus establish the well-posedness of the EHJB and the existence of equilibria in diffusion models under time-inconsistency, without resorting to conventional stringent regularity assumptions of the EHJB.