Active cyber defenses based on temporal platform diversity have been proposed as way to make systems more resistant to attacks. These defenses change the properties of the platforms in order to make attacks more complicated. Unfortunately, little work has been done on measuring the effectiveness of these defenses. In this work, we use four different approaches to quantitatively analyze these defenses; an abstract analysis studies the algebraic models of a temporal platform diversity system; a set of experiments on a test bed measures the metrics of interest for the system; a game theoretic analysis studies the impact of preferential selection of platforms and derives an optimal strategy; finally, a set of simulations evaluates the metrics of interest on the models. Our results from these approaches all agree and yet are counter-intuitive. We show that although platform diversity can mitigate some attacks, it can be detrimental for others. We also illustrate that the benefit from these systems heavily depends on their threat model and that the preferential selection of platforms can achieve better protection.