Ray tracing (RT) has emerged as a key tool for propagation channel modeling and network planning. Conventional RT is based on electromagnetic (EM) wave theory and its application relies on detailed mesh-based environment representations and material properties. In realistic environments, limited environmental geometry and material uncertainties hinder its scalability to complex scenarios. In this paper, we propose a novel physics aware neural RT surrogate named PointNeRT to address these limitations. The proposed model directly takes point clouds as environmental input, and efficiently reconstruct multipath without explicitly constructing mesh models or manually defining EM interaction rules. PointNeRT adopts a hop-by-hop modeling strategy guided by physical interaction constraints. It supports sequential prediction of multipath propagation and power attenuation. Numerical results and experiments demonstrate that the proposed method implicitly captures surface normal characteristics and EM material effects. It further achieves robust generalization in mobility scenarios and provides a physics-guided neural modeling of multipath propagation.