Interactive recommendation is able to learn from the interactive processes between users and systems to confront the dynamic interests of users. Recent advances have convinced that the ability of reinforcement learning to handle the dynamic process can be effectively applied in the interactive recommendation. However, the sparsity of interactive data may hamper the performance of the system. We propose to train a Model-agnostic Counterfactual Synthesis Policy to generate counterfactual data and address the data sparsity problem by modelling from observation and counterfactual distribution. The proposed policy can identify and replace the trivial components for any state in the training process with other agents, which can be deployed in any RL-based algorithm. The experimental results demonstrate the effectiveness and generality of our proposed policy.