Multiple hypothesis testing often involves composite nulls, i.e., nulls that are associated with two or more distributions. In many cases, it is reasonable to assume that there is a prior distribution on the distributions despite it is unknown. When the number of distributions under true nulls is finite, we show that under the above assumption, the false discover rate (FDR) can be controlled using $p$-values computed under constraints imposed by the empirical distribution of the observations. Comparing to FDR control using $p$-values defined as maximum significance level over all null distributions, the proposed FDR control can have substantially more power.