Fault Tree analysis is a widely used failure analysis methodology to assess a system in terms of safety or reliability in many industrial application domains. However, with Fault Tree methodology there is no possibility to express a temporal sequence of events or state-dependent behavior of software-controlled systems. In contrast to this, Markov Chains are a state-based analysis technique based on a stochastic model. But the use of Markov Chains for failure analysis of complex safety-critical systems is limited due to exponential explosion of the size of the model. In this paper, we present a concept to integrate Markov Chains in Component Fault Tree models. Based on a component concept for Markov Chains, which enables the association of Markov Chains to system development elements such as components, complex or software-controlled systems can be analyzed w.r.t. safety or reliability in a modular and compositional way. We illustrate this approach using a case study from the automotive domain.