This paper introduces a novel smartphone-enabled localization technology for ambient Internet of Things (IoT) devices, leveraging the widespread use of smartphones. By utilizing the passive movement of a smartphone, we create a virtual large array that enables direct localization using only angle-of-arrival (AoA) information. Unlike traditional two-step localization methods, direct localization is unaffected by AoA estimation errors in the initial step, which are often caused by multipath channels and noise. However, direct localization methods typically require prior environmental knowledge to define the search space, with calculation time increasing as the search space expands. To address limitations in current direct localization methods, we propose a machine learning (ML)-based direct localization technique. This technique combines ML with an adaptive matching pursuit procedure, dynamically generating search spaces for precise source localization. The adaptive matching pursuit minimizes location errors despite potential accuracy fluctuations in ML across various training and testing environments. Additionally, by estimating the reflection source's location, we reduce the effects of multipath channels, enhancing localization accuracy. Extensive three-dimensional ray-tracing simulations demonstrate that our proposed method outperforms current state-of-the-art direct localization techniques in computational efficiency and operates independently of prior environmental knowledge.