Cameras capture images that are essential for many safety-critical tasks. To process these images, a complex pipeline with multiple layers is used. Security attacks on this pipeline can severely affect passenger safety and system performance. However, many attacks presented in scientific literature overlook the fact that there are different layers and, hence, the feasibility and impact of these attacks can vary. While there has been research to improve the quality and robustness of the image processing pipeline, these efforts are often orthogonal to security research without exploiting potential overlap and synergies. In this work, we aim to bridge this gap by combining security and robustness research for the image processing pipeline in autonomous vehicles. We thoroughly investigated the body of literature on the security and robustness of the image processing pipeline and selected 92 papers for deeper discussion in this SoK. For the security domain, we classify the risk of attacks using the automotive security standard ISO 21434, emphasizing the need to consider all layers for overall system security. With our online tool TARA-CAM, we propose an interactive method to perform threat analysis and risk assessment following the ISO standard. We also demonstrate how existing robustness research can help mitigate the impact of attacks, addressing the current research gap. Finally, we present PICT, an embedded open-source testbed that can influence various parameters across all layers, allowing researchers to analyze the effects of different defense strategies and attack impacts. With this SoK, we contribute a comprehensive discussion and systematic analysis of existing approaches to image processing pipeline security and robustness, together with an open-source tool and testbed that jointly facilitates hardening the image processing pipeline against existing and future security attacks.