The proliferation of political memes in modern information campaigns calls for efficient solutions for image classification by ideological affiliation. While significant advances have recently been made on text classification in modern natural language processing literature, understanding the political insinuation in imagery is less developed due to the hard nature of the problem. Unlike text, where meaning arises from juxtaposition of tokens (words) within some common linguistic structures, image semantics emerge from a much less constrained process of fusion of visual concepts. Thus, training a model to infer visual insinuation is possibly a more challenging problem. In this paper, we explore an alternative unsupervised approach that, instead, infers ideological affiliation from image propagation patterns on social media. The approach is shown to improve the F1-score by over 0.15 (nearly 25%) over previous unsupervised baselines, and then by another 0.05 (around 7%) in the presence of a small amount of supervision.