If you’ve used image search on Google Photos, asked ChatGPT to describe a picture, or seen a model retrieve “a red bicycle parked on a city street” from a sea of unrelated photos, you’ve already seen contrastive learning at work. It’s the trick behind models like CLIP, and it’s quietly become one of the most useful ideas in modern ML. The cool part is that the underlying idea is genuinely simple. No fancy losses. No complex architectures. Just a clean question: can we teach a model that “this image” and “this caption” belong together, and everything else does not? In this post I want to walk through how that works for image–text pairs, end to end. The core idea Contrastive learning has exactly two moves: pull matching pairs together in a shared embedding space, and push non-matching pairs apart . That’s it. If you remember nothing else from this post, remember those two arrows. Two operations — pull and push — applied in a shared embedding space The reason this is powerful is that we don’t need labels in the traditional sense. We don’t need someone to tag every image with one of 1000 ImageNet classes. We just need pairs that go together , a photo and its caption, a question and its answer, a song and its lyrics. The web is full of those, which is why this approach scaled the way it did. What does the data look like? For image–text contrastive learning, the data is exactly what you’d hope: a bunch of (image, caption) pairs. To keep the running example concrete, here are four pairs we’ll use throughout the post: A tiny batch of image–text pairs. Each image has exactly one caption that belongs to it; everything else is a non-match The clever bit: when we feed a batch of N pairs into the model, we get N matches and N 2− N non-matches for free . With a batch of 4, that’s 4 positives and 12 negatives. With a batch of 32,768 (which is what CLIP used), it’s a lot more. Bigger batches → more negatives → more useful contrastive signal. Two encoders, one shared space Images and text are very different beasts. A 224×224 RGB image is a 150,528-dimensional integer tensor; a caption is a sequence of tokens. So we use two separate encoders — one for each modality — and have them both spit out a vector of the same size, say 512 dimensions. Two encoders, two output streams, but vectors of the same shape Typical choices are a ViT or ResNet for the image side and a Transformer for the text side, but really anything that produces a fixed-length vector will do. The important thing is that the two output spaces have the same dimensionality, because we're about to compare them directly. One detail that matters: we usually L2-normalize the embeddings before comparing them. That puts every vector on the unit hypersphere, which makes cosine similarity well-behaved and stops the model from cheating by just making vectors really long. Computing similarity Once both modalities live in the same space, comparing them is easy: take the dot product. If the vectors are L2-normalized, this is the cosine similarity, which lives in [−1,1]. We do this for every image–text pair in the batch, which gives us an N × N similarity matrix. The similarity matrix. The diagonal is what we want to push up; everything else is what we want to push down Look at the diagonal: 0.92, 0.94, 0.93. Those are the matching pairs, the dog with the dog caption, the lake with the lake caption. The model has learned that a cat photo doesn’t belong with a description of a mountain lake. Reframed this way, the training problem becomes: make the diagonal of this matrix as bright as possible, and the off-diagonal as dark as possible . Which leads us nicely into the loss. The contrastive objective Here’s the elegant move that makes everything click. Read each row of the similarity matrix as the logits of a classifier , where the correct class is the one on the diagonal. Suddenly we can use plain old cross-entropy. Positives in, negatives out. The loss is the bookkeeping that makes this happen For a single image i , the loss is: InfoNCE Loss The numerator rewards the model when image i is close to its own caption ti ​. The denominator sums over all N captions in the batch and acts like a normalizer, for the loss to go down, the matching pair has to “win” the softmax against all the non-matching ones. That’s the contrastive part. The τ (“temperature”) is a small positive number that controls how sharp the softmax is. Lower temperature = sharper = more aggressive about separating positives from hard negatives. CLIP makes τ a learnable parameter. We compute this loss in both directions — image-to-text and text-to-image — and average them. That symmetry is what makes the embedding space coherent from both sides. In code, the whole thing is shockingly compact: import torch import torch.nn.functional as F def contrastive_loss(image_embs, text_embs, temperature=0.07): # L2-normalize so dot product == cosine similarity image_embs = F.normalize(image_embs, dim=-1) text_embs = F.normalize(text_embs, dim=-1) # N x N similarity matrix, scaled by temperature logits = (image_embs @ text_embs.T) / temperature # The correct class for row i is column i — the diagonal labels = torch.arange(len(image_embs), device=logits.device) # Symmetric cross-entropy: image -> text and text -> image loss_i2t = F.cross_entropy(logits, labels) loss_t2i = F.cross_entropy(logits.T, labels) return (loss_i2t + loss_t2i) / 2 Six lines of real work. That’s the whole CLIP loss. What the embedding space looks like after training After enough iterations of “pull, push, pull, push,” something nice emerges: the embedding space organizes itself by meaning . Photos of dogs cluster near captions about dogs. Photos of mountains cluster near captions about mountains. And, this is the part that gets people excited, concepts that are semantically close end up close in space, even if you never explicitly told the model so. The embedding space after training. Each cluster is a concept; nearness encodes meaning This is what makes zero-shot classification work. To classify an image as “cat” or “dog” with a trained model, you don’t need to fine-tune anything. You just embed the image, embed the strings "a photo of a cat" and "a photo of a dog", and pick whichever caption is closer. The model never saw your specific labels in training, but because it learned to align language with images in general, the comparison just works. Putting it all together: the training loop Here is the entire pipeline in one picture, top to bottom: One training step. Repeat a few hundred million times and you get CLIP Each step: Sample a batch of N image–text pairs. Run images through the image encoder, captions through the text encoder. L2-normalize and form the N × N similarity matrix. Compute the symmetric contrastive loss against the diagonal. Backprop, update both encoders, repeat. That’s the whole algorithm. The hard parts are not in the loss, they’re in scale, data quality, and tuning. Takeaways Contrastive learning aligns two modalities (or two views, or two anything) in a shared embedding space. The objective is just: matching pairs close, non-matching pairs far. Cross-entropy on the similarity matrix’s diagonal makes that work. You get powerful zero-shot retrieval and classification almost for free, because the model learns meaning , not labels. The same recipe extends well beyond image–text — audio–text, video–text, code–doc, even single-modality self-supervision (SimCLR) all use this template. Thanks for reading — and as always, feel free to reach out if you have questions or want to chat about this stuff. Contrastive Learning was originally published in GoPenAI on Medium, where people are continuing the conversation by highlighting and responding to this story.