We address a single server queue control problem (QCP) in heavy traffic originating from Lee and Weerasinghe (2011). The state process represents the offered waiting time, the customer arrival has a state-dependent intensity, and the customers' service and patience times are i.i.d with general distributions. We introduce an infinite-horizon discounted cost functional consisting of a control cost generated from the use of control and a penalty for idleness cost. Our primary goal is to tackle the QCP, taking into account a non-trivial control cost and a non-increasing cost function resulting from the control mechanisms in the waiting time. Under mild assumptions, the heavy traffic limit of the QCP yields a stochastic control problem described by a diffusion process, which we call a diffusion control problem (DCP). We find the optimal control of the associated DCP by incorporating the Legendre-Fenchel transform and a formal Hamilton-Jacobi-Bellman (HJB) equation. Then, we ``translate'' this optimal strategy to the QCP, of which we obtain an asymptotically optimal policy. Apart from theoretical results, we also examine the REINFORCE algorithm, a Reinforcement learning (RL) approach, for solving stochastic controls motivated by recent literature. We highlight the advantages and limitations of simulation from theoretical results and data-driven algorithms.