We consider cross-layer design of delay optimal transmission strategies for energy harvesting transmitters where the data and energy arrival processes are stochastic. Using Markov decision theory, we show that the value function is weakly increasing in the queue state and weakly decreasing in the battery state. It is natural to expect that the delay optimal policy should be weakly increasing in the queue and battery states. We show via counterexamples that this is not the case. In fact, we show that for some sample scenarios the delay optimal policy may perform 5-13% better than the best monotone policy.