We study the problem of clock synchronization in a networked system with arbitrary starts for all nodes. We consider a synchronous network of $n$ nodes, where each node has a local clock that is an integer counter. Eventually, clocks must be all equal and increase by one in each round modulo some period $P$. The purpose of this paper is to study whether clock synchronization can be achieved with bounded memory, that is every node maintains a number of states that does not depend on the network size. In particular, we are interested in clock synchronization algorithms which work in dynamic networks, i.e., tolerate that communication links continuously fail and come-up. We first focus on self-stabilizing solutions for clock synchronization, and prove that there is no such algorithm that is bounded memory, even in the case of static networks. More precisely, we show a lower bound of $n+1$ states at each node required to achieve clock synchronization in static strongly connected networks with at most $n$ nodes, and derive a lower bound of $n-2$ rounds on synchronization time, in the worst case. We then prove that, when the self-stabilizing requirement is removed, the impossibility of clock synchronization with bounded memory still holds in the dynamic setting: every solution for the clock synchronization problem in dynamic networks with at most $n$ nodes requires each node to have $Ω(\log n)$ states.