Programmable radio environments parametrized by reconfigurable intelligent surfaces (RISs) are emerging as a new wireless communications paradigm, but currently used channel models for the design and analysis of signal-processing algorithms cannot include fading in a manner that is faithful to the underlying wave physics. To overcome this roadblock, we introduce a physics-based end-to-end model of RIS-parametrized wireless channels with adjustable fading (coined PhysFad) which is based on a first-principles coupled-dipole formalism. PhysFad naturally incorporates the notions of space and causality, dispersion (i.e., frequency selectivity) and the intertwinement of each RIS element's phase and amplitude response, as well as any arising mutual coupling effects including long-range mesoscopic correlations. PhysFad offers the to-date missing tuning knob for adjustable fading. We thoroughly characterize PhysFad and demonstrate its capabilities for a prototypical problem of RIS-enabled over-the-air channel equalization in rich-scattering wireless communications. We also share a user-friendly version of our code to help the community transition towards physics-based models with adjustable fading.