This paper presents an approach to the evaluation and validation of mass spectrometry data for construction of an `early warning' diagnostic procedure. We describe implementation of a designed experiment and place emphasis on the consistent and correct use of validation based evaluation - which is a key requirement to achieve unbiased assessment of the ability of mass spectrometry data for diagnosis in this setting. Strict adherence to validation as a scientific principle will however typically imply that the analyst must make choices. Like all choices in statistical analysis, validation comes at a cost! We present a detailed and extensive discussion of the issues involved and propose that much greater emphasis and requirement for validation would enter clinical proteomic science.