View PDF HTML (experimental)

Abstract:Rate-independent sequence models respond to the ordered structure of input extrema rather than to absolute timing or token position. This principle underlies Preisach Attention, a hysteretic alternative to softmax attention in which sequence history is represented by a stack of alternating extrema generated by the classical wiping-out rule. This paper establishes the information-theoretic and online algorithmic foundations of that representation.
We prove that the Preisach extremum stack is not merely a convenient implementation detail, but the complete invariant of computable rate-independent sequence functionals: a functional is rate-independent if and only if it factors through the stack. We then show that the stack is minimal in two complementary senses. In the Kolmogorov setting, the shortest exact representation answering all rate-independent queries has complexity equal to that of the stack up to an additive constant independent of sequence length and stack depth. In the Shannon setting, under any input distribution, every sufficient representation contains at least as much mutual information about the input as the stack, with equality only for representations informationally equivalent to it.
Finally, we analyse the online maintenance cost of this minimal state. Although the standard stack update is amortised constant time, adversarial inputs can induce linear worst-case latency. We prove a matching output-change lower bound, show that binary search reduces boundary detection but not deletion, and give an exact finger-tree implementation with worst-case (O(\log k)) update time, where (k) is the current stack depth. The results provide a principled foundation for replacing full sequence histories or KV-cache-like memories by extremum-stack states in rate-independent neural architectures, without approximation and with bounded online latency.

Submission history

From: Piotr Frydrych Ph. D. Eng. [view email]
[v1] Wed, 3 Jun 2026 11:23:57 UTC (26 KB)
[v2] Mon, 13 Jul 2026 10:17:02 UTC (27 KB)