High resolution computed tomography (HRCT) is the most important imaging modality for interstitial lung diseases, where the radiologists are interested in identifying certain patterns, and their volumetric and regional distribution. The use of machine learning can assist the radiologists with both these tasks by performing semantic segmentation. In this paper, we propose an interactive annotation-tool for semantic segmentation that assists the radiologist in labeling CT scans. The annotation tool is evaluated by six radiologists and radiology residents classifying healthy lung and reticular pattern i HRCT images. The usability of the system is evaluated with a System Usability Score (SUS) and interaction information from the readers that used the tool for annotating the CT volumes. It was discovered that the experienced usability and how the users interactied with the system differed between the users. A higher SUS-score was given by users that prioritized learning speed over model accuracy and spent less time with manual labeling and instead utilized the suggestions provided by the GUI. An analysis of the annotation variations between the readers show substantial agreement (Cohen's kappa=0.69) for classification of healthy and affected lung parenchyma in pulmonary fibrosis. The inter-reader variation is a challenge for the definition of ground truth.