Ambient energy harvesting is touted as a low cost solution to prolong the life of low-powered devices, reduce the carbon footprint, and make the system self-sustainable. Most research to date have focused either on the physical aspects of energy conversion process or on optimal consumption policy of the harvested energy at the system level. However, although intuitively understood, to the best of our knowledge, the idea of self-sustainability is yet to be made precise and studied as a performance metric. In this paper, we provide a mathematical definition of the concept of self-sustainability of an energy harvesting system, based on the complementary idea of eventual outage. In particular, we analyze the harvest-store-consume system with infinite battery capacity, stochastic energy arrivals, and fixed energy consumption rate. Using the random walk theory, we identify the necessary condition for the system to be self-sustainable. General formulas are given for the self-sustainability probability in the form of integral equations. Since these integral equations are difficult to solve analytically, an exponential upper bound for eventual outage probability is given using martingales. This bound guarantees that the eventual outage probability can be made arbitrarily small simply by increasing the initial battery energy. We also give an asymptotic formula for eventual outage. For the special case when the energy arrival follows a Poisson process, we are able to find the exact formulas for the eventual outage probability. We also show that the harvest-store-consume system is mathematically equivalent to a $GI/G/1$ queueing system, which allows us to easily find the outage probability, in case the necessary condition for self-sustainability is violated. Monte-Carlo simulations verify our analysis.