Cooperative Adaptive Cruise Control (CACC) is one of the driving applications of vehicular ad-hoc networks (VANETs) and promises to bring more efficient and faster transportation through cooperative behavior between vehicles. In CACC, vehicles exchange information, which is relied on to partially automate driving; however, this reliance on cooperation requires resilience against attacks and other forms of misbehavior. In this paper, we propose a rigorous attacker model and an evaluation framework for this resilience by quantifying the attack impact, providing the necessary tools to compare controller resilience and attack effectiveness simultaneously. Although there are significant differences between the resilience of the three analyzed controllers, we show that each can be attacked effectively and easily through either jamming or data injection. Our results suggest a combination of misbehavior detection and resilient control algorithms with graceful degradation are necessary ingredients for secure and safe platoons.