A classic approach for solving differential equations with neural networks builds upon neural forms, which employ the differential equation with a discretisation of the solution domain. Making use of neural forms for time-dependent differential equations, one can apply the recently developed method of domain fragmentation. That is, the domain may be split into several subdomains, on which the optimisation problem is solved. In classic adaptive numerical methods, the mesh as well as the domain may be refined or decomposed, respectively, in order to improve accuracy. Also the degree of approximation accuracy may be adapted. It would be desirable to transfer such important and successful strategies to the field of neural network based solutions. In the present work, we propose a novel adaptive neural approach to meet this aim for solving time-dependent problems. To this end, each subdomain is reduced in size until the optimisation is resolved up to a predefined training accuracy. In addition, while the neural networks employed are by default small, we propose a means to adjust also the number of neurons in an adaptive way. We introduce conditions to automatically confirm the solution reliability and optimise computational parameters whenever it is necessary. Results are provided for several initial value problems that illustrate important computational properties of the method alongside. In total, our approach not only allows to analyse in high detail the relation between network error and numerical accuracy. The new approach also allows reliable neural network solutions over large computational domains.