Seismic events, among many other natural hazards, reduce due functionality and exacerbate vulnerability of in-service buildings. Accurate modeling and prediction of building's response subjected to earthquakes makes possible to evaluate building performance. To this end, we leverage the recent advances in deep learning and develop a physics-guided convolutional neural network (PhyCNN) framework for data-driven seismic response modeling and serviceability assessment of buildings. The proposed PhyCNN approach is capable of accurately predicting building's seismic response in a data-driven fashion without the need of a physics-based analytical/numerical model. The basic concept is to train a deep PhyCNN model based on available seismic input-output datasets (e.g., from simulation or sensing) and physics constraints. The trained PhyCNN can then used as a surrogate model for structural seismic response prediction. Available physics (e.g., the law of dynamics) can provide constraints to the network outputs, alleviate overfitting issues, reduce the need of big training datasets, and thus improve the robustness of the trained model for more reliable prediction. The trained surrogate model is then utilized for fragility analysis given certain limit state criteria (e.g., the serviceability state). In addition, an unsupervised learning algorithm based on K-means clustering is also proposed to partition the limited number of datasets to training, validation and prediction categories, so as to maximize the use of limited datasets. The performance of the proposed approach is demonstrated through three case studies including both numerical and experimental examples. Convincing results illustrate that the proposed PhyCNN paradigm outperforms conventional pure data-based neural networks.