Neural Networks (NNs) have been identified as a potentially powerful tool in the study of complex dynamical systems. A good example is the NN differential equation (DE) solver, which provides closed form, differentiable, functional approximations for the evolution of a wide variety of dynamical systems. A major disadvantage of such NN solvers can be the amount of computational resources needed to achieve accuracy comparable to existing numerical solvers. We present new strategies for existing dynamical system NN DE solvers, making efficient use of the \textit{learnt} information, to speed up their training process, while still pursuing a completely unsupervised approach. We establish a fundamental connection between NN theory and dynamical systems theory via Koopman Operator Theory (KOT), by showing that the usual training processes for Neural Nets are fertile ground for identifying multiple Koopman operators of interest. We end by illuminating certain applications that KOT might have for NNs in general.