Narrowing the performance gap between optimal and feasible detection in inter-symbol interference (ISI) channels, this paper proposes to use graph neural networks (GNNs) for detection that can also be used to perform joint detection and decoding (JDD). For detection, the GNN is build upon the factor graph representations of the channel, while for JDD, the factor graph is expanded by the Tanner graph of the parity-check matrix (PCM) of the channel code, sharing the variable nodes (VNs). A particularly advantageous property of the GNN is a) the robustness against cycles in the factor graphs which is the main problem for sum-product algorithm (SPA)-based detection, and b) the robustness against channel state information (CSI) uncertainty at the receiver. Additionally, we propose using an input embedding resulting in a GNN independent of the channel impulse response (CIR). Consequently, a fully deep learning-based receiver enables joint optimization instead of individual optimization of the components, so-called end-to-end learning. Furthermore, we propose a parallel flooding schedule that also reduces the latency, which turns out to improve the error correcting performance. The proposed approach is analyzed and compared to state-of-the-art baselines for different modulations and codes in terms of error correcting capability and latency. The gain compared to SPA-based detection might be explained with improved messages between nodes and adaptive damping of messages. For a higher order modulation in a high-rate turbo detection and decoding (TDD) scenario the GNN shows a, at first glance, surprisingly high gain of 6.25 dB compared to the best, feasible non-neural baseline.