The automatic assessment of health-related acoustic cues has the potential to improve healthcare accessibility and affordability. Although parametric models are promising, they face challenges in privacy and adaptability. To address these, we propose a NoN-Parametric framework for Speech-based symptom Assessment (NoNPSA). By isolating medical data in a retrieval datastore, NoNPSA avoids encoding private information in model parameters and enables efficient data updates. A self-supervised learning (SSL) model pre-trained on general-purpose datasets extracts features, which are used for similarity-based retrieval. Metadata-aware refinement filters the retrieved data, and associated labels are used to compute an assessment score. Experimental results show that NoNPSA achieves competitive performance compared to fine-tuning SSL-based methods, while enabling greater privacy, update efficiency, and adaptability--showcasing the potential of non-parametric approaches in healthcare.