Objective. Arrhythmia classification from electrocardiograms (ECGs) suffers from high false positive rates and limited cross-dataset generalization, particularly for atrial fibrillation (AF) detection where specificity ranges from 0.72 to 0.98 using conventional 30-s analysis windows. While most deep learning approaches analyze isolated 30-s ECG windows, many arrhythmias, including AF and atrial flutter, exhibit diagnostic features that emerge over extended time scales. Approach. We introduce S4ECG, a deep learning architecture based on structured state-space models (S4), designed to capture long-range temporal dependencies by jointly analyzing multiple consecutive ECG windows spanning up to 20 min. We evaluate S4ECG on four publicly available databases for multi-class arrhythmia classification and perform systematic cross-dataset evaluations to assess out-of-distribution robustness. Results. Multi-window analysis consistently outperforms single-window approaches across all datasets, improving macro-averaged AUROC by 1.0-11.6 percentage points. For AF, specificity increases from 0.718-0.979 to 0.967-0.998 at a fixed sensitivity threshold, yielding a 3-10-fold reduction in false positive rates. Significance. Compared with convolutional neural network baselines, the S4 architecture shows superior performance, and multi-window training substantially reduces cross-dataset degradation. Optimal diagnostic windows are 10-20 min, beyond which performance plateaus or degrades. These findings demonstrate that structured incorporation of extended temporal context enhances both arrhythmia classification accuracy and cross-dataset robustness. The identified optimal temporal windows provide practical guidance for ECG monitoring system design and may reflect underlying physiological timescales of arrhythmogenic dynamics.