We present a fast, differentiable, GPU-accelerated optimization method for ray path tracing in environments containing planar reflectors and straight diffraction edges. Based on Fermat's principle, our approach reformulates the path-finding problem as the minimization of total path length, enabling efficient parallel execution on modern GPU architectures. Unlike existing methods that require separate algorithms for reflections and diffractions, our unified formulation maintains consistent problem dimensions across all interaction sequences, making it particularly suitable for vectorized computation. Through implicit differentiation, we achieve efficient gradient computation without differentiating through solver iterations, significantly outperforming traditional automatic differentiation approaches. Numerical simulations demonstrate convergence rates comparable to specialized Newton methods while providing superior scalability for large-scale applications. The method integrates seamlessly with differentiable programming libraries such as JAX and DrJIT, enabling new possibilities in inverse design and optimization for wireless propagation modeling. The source code is openly available at https://github.com/jeertmans/fpt-jax.