Author(s): Utkarsh Mittal

Originally published on Towards AI.

Part 12 — https://medium.com/p/75cf0a345156

The article explains how to design a production recommender system using a real end-to-end scenario and concrete latency, data, and training considerations. It argues that business objectives differ from what can be directly labeled, and that ranking (not simple classification) with measurable proxy signals is central. It outlines what the system must do and must never do under tight latency constraints, why scale forces a two-stage architecture (fast retrieval followed by richer ranking), and how cold-start and feedback sparsity shape training data. It covers how labels are constructed, how negatives and time-based splits avoid bias and leakage, how feature stores prevent training-serving skew, and how two-tower retrieval with dot products and softmax training works in practice. It then discusses ranking with baselines like LightGBM and richer wide & deep models, calibration and multi-task refinements, and evaluation using Recall@K for retrieval and NDCG@K for ranking with error slicing. Finally, it walks through the live 200ms execution pipeline, operational optimizations (quantization, batching, caching, fallbacks, shadow mode), online evaluation methods (A/B bucketing, interleaving), and the four major ways such systems “rot” via monitoring gaps, popularity spirals, offline-online mismatch, and training-serving skew—closing with what distinguishes mid-level, senior, and staff engineering work in recommender systems.

Read the full blog for free on Medium.

Published via Towards AI