We consider Hodgkin-Huxley-type model that is a stiff ODE system with two fast and one slow variables. For the parameter ranges under consideration the original version of the model has unstable fixed point and the oscillating attractor that demonstrates bifurcation from bursting to spiking dynamics. Also a modified version is considered where the bistability occurs such that an area in the parameter space appears where the fixed point becomes stable and coexists with the bursting attractor. For these two systems we create artificial neural networks that are able to reproduce their dynamics. The created networks operate as recurrent maps and are trained on trajectory cuts sampled at random parameter values within a certain range. Although the networks are trained only on oscillatory trajectory cuts, it also discover the fixed point of the considered systems. The position and even the eigenvalues coincide very well with the fixed point of the initial ODEs. For the bistable model it means that the network being trained only on one brunch of the solutions recovers another brunch without seeing it during the training. These results, as we see it, are able to trigger the development of new approaches to complex dynamics reconstruction and discovering. From the practical point of view reproducing dynamics with the neural network can be considered as a sort of alternative method of numerical modeling intended for use with contemporary parallel hard- and software.