One dimensional excited random walk has been extensively studied for bounded, i.i.d. cookie environments. In this case, many important properties of the walk including transience or recurrence, positivity or non-positivity of the speed, and the limiting distribution of the position of the walker are all characterized by a single parameter $δ$, the total expected drift per site. In the more general case of stationary ergodic environments, things are not so well understood. If all cookies are positive then the same threshold for transience vs. recurrence holds, even if the cookie stacks are unbounded. However, it is unknown if the threshold for transience vs. recurrence extends to the case when cookies may be negative (even for bounded stacks), and moreover there are simple counterexamples to show that the threshold for positivity of the speed does not. It is thus natural to study the behavior of the model in the case of Markovian environments, which are intermediate between the i.i.d. and stationary ergodic cases. We show here that many of the important results from the i.i.d. setting, including the thresholds for transience and positivity of the speed, as well as the limiting distribution of the position of the walker, extend to a large class of Markovian environments. No assumptions are made about the positivity of the cookies.