Autonomous vehicles often perceive the environment by feeding sensor data to a learned detector algorithm, then feeding detections to a multi-object tracker that models object motions over time. Probabilistic models of multi-object trackers typically assume that errors in the detector algorithm occur randomly over time. We instead assume that undetected objects and false detections will persist in certain conditions, and modify the tracking framework to account for them. The modifications are tested on a vehicle tracking dataset using a state-of-the-art lidar-based detector, a novel lightweight detector, and a fusion of camera and lidar detectors. For each detector, the persistence modifications notably improve performance and enable the model to outperform baseline trackers.