Ascent sequences play a key role in the combinatorics of Fishburn structures. Difference ascent sequences are a natural generalization obtained by replacing ascents with $d$-ascents. We have recently extended the so-called hat map to difference ascent sequences, and self-modified difference ascent sequences are the fixed points under this map. We characterize self-modified difference ascent sequences and enumerate them in terms of certain generalized Fibonacci polynomials. Furthermore, we describe the corresponding subset of $d$-Fishburn permutations.