One of the focus areas of modern scientific research is to reveal mysteries related to genes and their interactions. The dynamic interactions between genes can be encoded into a gene regulatory network (GRN), which can be used to gain understanding on the genetic mechanisms behind observable phenotypes. GRN inference from time series data has recently been a focus area of systems biology. Due to low sampling frequency of the data, this is a notoriously difficult problem. We tackle the challenge by introducing the so-called continuous-time Gaussian process dynamical model, based on Gaussian process framework that has gained popularity in nonlinear regression problems arising in machine learning. The model dynamics are governed by a stochastic differential equation, where the dynamics function is modelled as a Gaussian process. We prove the existence and uniqueness of solutions of the stochastic differential equation. We derive the probability distribution for the Euler discretised trajectories and establish the convergence of the discretisation. We develop a GRN inference method called BINGO, based on the developed framework. BINGO is based on MCMC sampling of trajectories of the GPDM and estimating the hyperparameters of the covariance function of the Gaussian process. Using benchmark data examples, we show that BINGO is superior in dealing with poor time resolution and it is computationally feasible.