When evolution algorithms are used to unfold the neutron energy spectrum, fitness function design is an important fundamental work for evaluating the quality of the solution, but it has not attracted much attention. In this work, we investigated the performance of eight fitness functions attached to the genetic algorithm (GA) and the differential evolution algorithm (DEA) used for unfolding four neutron spectra selected from the IAEA 403 report. Experiments show that the fitness functions with a maximum in the GA can limit the ability of the population to percept the fitness change, but the ability can be made up in the DEA. The fitness function with a feature penalty term helps to improve the performance of solutions, and the fitness function using the standard deviation and the Chi-squared result shows the balance between the algorithm and the spectra. The results also show that the DEA has good potential for neutron energy spectrum unfolding. The purposes of this work are to provide evidence for structuring and modifying the fitness functions and to suggest some genetic operations that should receive attention when using the fitness function to unfold neutron spectra.