This paper proposes and studies a general form of dynamic $N$-player non-cooperative games called $α$-potential games, where the change of a player's value function upon her unilateral deviation from her strategy is equal to the change of an $α$-potential function up to an error $α$. Analogous to the static potential game (which corresponds to $α=0$), the $α$-potential game framework is shown to reduce the challenging task of finding $α$-Nash equilibria for a dynamic game to minimizing the $α$-potential function. Moreover, an analytical characterization of $α$-potential functions is established, with $α$ represented in terms of the magnitude of the asymmetry of value functions' second-order derivatives. For stochastic differential games in which the state dynamic is a controlled diffusion, $α$ is characterized in terms of the number of players, the choice of admissible strategies, and the intensity of interactions and the level of heterogeneity among players. Two classes of stochastic differential games, namely distributed games and games with mean field interactions, are analyzed to highlight the dependence of $α$ on general game characteristics that are beyond the mean-field paradigm, which focuses on the limit of $N$ with homogeneous players. To analyze the $α$-NE, the associated optimization problem is embedded into a conditional McKean-Vlasov control problem. A verification theorem is established to construct $α$-NE based on solutions to an infinite-dimensional Hamilton-Jacobi-Bellman equation, which is reduced to a system of ordinary differential equations for linear-quadratic games.