In this work we study the secrecy capacity of Gaussian multiple-input multiple-output (MIMO) wiretap channels (WTCs) with a finite memory, subject to a per-symbol average power constraint on the MIMO channel input. MIMO channels with finite memory are very common in wireless communications as well as in wireline communications (e.g., in communications over power lines). To derive the secrecy capacity of the Gaussian MIMO WTC with finite memory we first construct an asymptotically-equivalent block-memoryless MIMO WTC, which is then transformed into a set of parallel, independent, memoryless MIMO WTCs in the frequency domain. The secrecy capacity of the Gaussian MIMO WTC with finite memory is obtained as the secrecy capacity of the set of parallel independent memoryless MIMO WTCs, and is expressed as a maximization over the input covariance matrices in the frequency domain. Lastly, we detail two applications of our result: First, we show that the secrecy capacity of the Gaussian scalar WTC with finite memory can be achieved by waterfilling, and obtain a closed-form expression for this secrecy capacity. Then, we use our result to characterize the secrecy capacity of narrowband powerline channels, thereby resolving one of the major open issues for this channel model.