Novikov algebras provide a simple but powerful algebraic axiomatization of important features of classical diferential calculus. We study their structure properties, modeling their relationships with commutative algebras with a derivation, featuring the role of their Lie and pre-Lie structures and analyzing the structure of their enveloping algebras. We focus on the combinatorial analysis of the Poincaré-Birkhoff-Witt Theorem (classical and pre-Lie), the pre-Lie exponential and logarithm. The topic is important for applications of the theory and has been treated intensively for pre-Lie algebras. However, specific formulas can be obtained in the Novikov case. We analyze their structure, as well as featuring various remarkable properties. Related statistical phenomena on trees, tableaux and permutations are investigated in this context.