Statistical inference for extreme values of random events is difficult in practice due to low sample sizes and inaccurate models for the studied rare events. If prior knowledge for extreme values is available, Bayesian statistics can be applied to reduce the sample complexity, but this requires a known probability distribution. By working with the quantiles for extremely low probabilities (in the order of $10^{-2}$ or lower) and relying on their asymptotic normality, inference can be carried out without assuming any distributions. Despite relying on asymptotic results, it is shown that a Bayesian framework that incorporates prior information can reduce the number of observations required to estimate a particular quantile to some level of accuracy.