In this paper the class of ARCH$(\infty)$ models is generalized to the nonstationary class of ARCH$(\infty)$ models with time-varying coefficients. For fixed time points, a stationary approximation is given leading to the notation ``locally stationary ARCH$(\infty)$ process.'' The asymptotic properties of weighted quasi-likelihood estimators of time-varying ARCH$(p)$ processes ($p<\infty$) are studied, including asymptotic normality. In particular, the extra bias due to nonstationarity of the process is investigated. Moreover, a Taylor expansion of the nonstationary ARCH process in terms of stationary processes is given and it is proved that the time-varying ARCH process can be written as a time-varying Volterra series.