Progressive quenching (PQ) is the stochastic process in which the system's degrees of freedom are sequentially fixed. While such process does not satisfy the local detailed balance, it has been found that the some physical observable of a complete spin network exhibits the martingale property. We studied system's response to the perturbation given at intermediate stages of the PQ. The response at the final stage reveals the persistent memory, and we show that this persistence is a direct consequence of the martingale process behind. Not only the mean response, the shape of the probability distribution at the stage of perturbation is also memorized. Using the hidden martingale process we can predict the final bimodal distribution from the early-stage unimodal distribution in the regime where the unfrozen spins are paramagnetic. We propose a viewpoint that the martingale property is a stochastic conservation law which is supported behind by some stochastic invariance.