Institutions can provide incentives to increase cooperation behaviour in a population where this behaviour is infrequent. This process is costly, and it is thus important to optimize the overall spending. This problem can be mathematically formulated as a multi-objective optimization problem where one wishes to minimize the cost of providing incentives while ensuring a desired level of cooperation within the population. In this paper, we provide a rigorous analysis for this problem. We study cooperation dilemmas in both the pairwise (the Donation game) and multi-player (the Public Goods game) settings. We prove the regularity of the (total incentive) cost function, characterize its asymptotic limits (infinite population, weak selection and large selection) and show exactly when reward or punishment is more efficient. We prove that the cost function exhibits a phase transition phenomena when the intensity of selection varies. We calculate the critical threshold in regards to the phase transition and study the optimization problem when the intensity of selection is under and above the critical value. It allows us to provide an exact calculation for the optimal cost of incentive, for a given intensity of selection. Finally, we provide numerical simulations to demonstrate the analytical results. Overall, our analysis provides for the first time a selection-dependent calculation of the optimal cost of institutional incentives (for both reward and punishment) that guarantees a minimum amount of cooperation. It is of crucial importance for real-world applications of institutional incentives since intensity of selection is specific to a given population and the underlying game payoff structure.