Fault diagnosis of dynamic systems is done by detecting changes in time-series data, for example residuals, caused by system degradation and faulty components. The use of general-purpose multi-class classification methods for fault diagnosis is complicated by imbalanced training data and unknown fault classes. Another complicating factor is that different fault classes can result in similar residual outputs, especially for small faults, which causes classification ambiguities. In this work, a framework for data-driven analysis and open-set classification is developed for fault diagnosis applications using the Kullback-Leibler divergence. A data-driven fault classification algorithm is proposed which can handle imbalanced datasets, class overlapping, and unknown faults. In addition, an algorithm is proposed to estimate the size of the fault when training data contains information from known fault realizations. An advantage of the proposed framework is that it can also be used for quantitative analysis of fault diagnosis performance, for example, to analyze how easy it is to classify faults of different magnitudes. To evaluate the usefulness of the proposed methods, multiple datasets from different fault scenarios have been collected from an internal combustion engine test bench to illustrate the design process of a data-driven diagnosis system, including quantitative fault diagnosis analysis and evaluation of the developed open set fault classification algorithm.