The boundary element method (BEM) enables solving three-dimensional electromagnetic problems using a two-dimensional surface mesh, making it appealing for applications ranging from electrical interconnect analysis to the design of metasurfaces. The BEM typically involves the electric and magnetic fields as unknown quantities. Formulations based on electromagnetic potentials rather than fields have garnered interest recently, for two main reasons: (a) they are inherently stable at low frequencies, unlike many field-based approaches, and (b) potentials provide a more direct interface to quantum physical phenomena. Existing potential-based formulations for electromagnetic scattering have been proposed primarily for perfect conductors. We develop a potential-based BEM formulation which can capture both dielectric and conductive losses, and accurately models the skin effect over broad ranges of frequency. The accuracy of the proposed formulation is validated through canonical and realistic numerical examples.