Accurate prediction of fruit sugar content is essential for quality control and market valuation in agriculture. Conventional measurement techniques rely on destructive, time-consuming processes (e.g., juicing and refractometry) or direct contact instruments, which hinder high-throughput operations. This paper introduces SweetFruit, a mobile two-stage system that leverages low-cost sensors to estimate fruit sugar content without contact. In Stage 1, we implement a lightweight 3D deep learning model (SF-PointNet) that uses point clouds from a Time-of-Flight (ToF) depth camera to classify fruit as high or low sugar. In Stage 2, a regression network (SF-Net) predicts the fruit's Brix value using measurements from a compact 18-channel near-infrared (NIR) spectrometer. The system uses simple off-the-shelf sensors (AS7265x NIR and Arducam ToF) with efficient processing pipelines for real-time execution on embedded platforms. Experiments on green 'Granny Smith' apples and strawberries demonstrate the system's effectiveness. Stage 1 achieves over 90% classification accuracy, enabling rapid prescreening, while Stage 2 delivers precise sugar estimates, with a root mean square error (RMSE) of 0.57 Brix, reducing error by 22% compared to using NIR sensing alone. SweetFruit offers a scalable, field-ready solution for rapid fruit quality screening, showcasing the benefits of task-specific multimodal sensing in mobile agricultural applications.