We present an efficient machine learning framework for detection and classification of nanoparticles on surfaces that are detected in the far-field with Coherent Fourier Scatterometry (CFS). We study silicon wafers contaminated with spherical polystyrene (PSL) nanoparticles (with diameters down to $λ/8$). Starting from the raw data, the proposed framework does the pre-processing and particle search. Further, the unsupervised clustering algorithms, such as K-means and DBSCAN, are customized to be used to define the groups of signals that are attributed to a single scatterer. Finally, the particle count versus particle size histogram is generated. The challenging cases of the high density of scatterers, noise and drift in the dataset are treated. We take advantage of the prior information on the size of the scatterers to minimize the false-detections and as a consequence, provide higher discrimination ability and more accurate particle counting. Numerical and real experiments are conducted to demonstrate the performance of the proposed search and cluster-assessment techniques. Our results illustrate that the proposed algorithm can detect surface contaminants correctly and effectively.