In this paper, we consider the phase retrieval problem in which one aims to recover a signal from the magnitudes of affine measurements. Let $\{{\mathbf a}_j\}_{j=1}^m \subset {\mathbb H}^d$ and ${\mathbf b}=(b_1, \ldots, b_m)^\top\in{\mathbb H}^m$, where ${\mathbb H}={\mathbb R}$ or ${\mathbb C}$. We say $\{{\mathbf a}_j\}_{j=1}^m$ and $\mathbf b$ are affine phase retrievable for ${\mathbb H}^d$ if any ${\mathbf x}\in{\mathbb H}^d$ can be recovered from the magnitudes of the affine measurements $\{|<{\mathbf a}_j,{\mathbf x}>+b_j|,\, 1\leq j\leq m\}$. We develop general framework for affine phase retrieval and prove necessary and sufficient conditions for $\{{\mathbf a}_j\}_{j=1}^m$ and $\mathbf b$ to be affine phase retrievable. We establish results on minimal measurements and generic measurements for affine phase retrieval as well as on sparse affine phase retrieval. In particular, we also highlight some notable differences between affine phase retrieval and the standard phase retrieval in which one aims to recover a signal $\mathbf x$ from the magnitudes of its linear measurements. In standard phase retrieval, one can only recover $\mathbf x$ up to a unimodular constant, while affine phase retrieval removes this ambiguity. We prove that unlike standard phase retrieval, the affine phase retrievable measurements $\{{\mathbf a}_j\}_{j=1}^m$ and $\mathbf b$ do not form an open set in ${\mathbb H}^{m\times d}\times {\mathbb H}^m$. Also in the complex setting, the standard phase retrieval requires $4d-O(\log_2d)$ measurements, while the affine phase retrieval only needs $m=3d$ measurements.