The reliability function of memoryless channels with noiseless feedback and variable-length coding has been found to be a linear function of the average rate in the classic work of Burnashev. In this work we consider unifilar channels with noiseless feedback and study upper bounds for the channel reliability function with variable length codes. In unifilar channels the channel state is known to the transmitter but is unknown to the receiver. We generalize Burnashev's analysis and derive a similar expression which is linear in average rate and depends on the channel capacity, as well as an additional parameter which relates to a sequential binary hypothesis testing problem over this channel. This parameter is evaluated by setting up an appropriate Markov decision process (MDP). Furthermore, an upper bound for this parameter is derived using a simplified MDP. Numerical evaluation of the parameter for several binary input/state/output unifilar channels hints at the optimal transmission strategies. Such strategies are studied in a companion paper to provide lower (achievable) bounds on the channel reliability function.