Backdoor watermarking is a promising paradigm to protect the copyright of deep neural network (DNN) models. In the existing works on this subject, researchers have intensively focused on watermarking robustness, while the concept of fidelity, which is concerned with the preservation of the model's original functionality, has received less attention. In this paper, focusing on deep image classification models, we show that the existing shared notion of the sole measurement of learning accuracy is inadequate to characterize backdoor fidelity. Meanwhile, we show that the analogous concept of embedding distortion in multimedia watermarking, interpreted as the total weight loss (TWL) in DNN backdoor watermarking, is also problematic for fidelity measurement. To address this challenge, we propose the concept of deep fidelity, which states that the backdoor watermarked DNN model should preserve both the feature representation and decision boundary of the unwatermarked host model. To achieve deep fidelity, we propose two loss functions termed penultimate feature loss (PFL) and softmax probability-distribution loss (SPL) to preserve feature representation, while the decision boundary is preserved by the proposed fix last layer (FixLL) treatment, inspired by the recent discovery that deep learning with a fixed classifier causes no loss of learning accuracy. With the above designs, both embedding from scratch and fine-tuning strategies are implemented to evaluate the deep fidelity of backdoor embedding, whose advantages over the existing methods are verified via experiments using ResNet18 for MNIST and CIFAR-10 classifications, and wide residual network (i.e., WRN28_10) for CIFAR-100 task. PyTorch codes are available at https://github.com/ghua-ac/dnn_watermark.