Kinetic plasma simulations solve the Vlasov-Poisson or Vlasov-Maxwell equations to evolve scalar-variable distribution functions in position-velocity phase space and vector-variable electromagnetic fields in physical space. The computational cost of evolving high-dimensional variables often limits the utility of continuum kinetic simulations and presents a challenge when it comes to accurately simulating real-world physical phenomena. To address this challenge, we developed techniques that accelerate and minimize the computational work required for a scalable Vlasov-Poisson solver. We present theoretical hardware compute and communication bounds required for solving a fourth-order finite-volume Vlasov-Poisson system. These bounds are then used to inform and evaluate the design of performance portable algorithms for a multiple graphics processing unit (GPU) accelerated version of the Vlasov-Poisson solver VCK-CPU. We demonstrate that the multi-GPU Vlasov solver implementation VCK-GPU simultaneously minimizes required inter-process data transfer while also being bounded by the machine network performance limits, leaving minimal room for theoretical performance improvements. This resulted in an overall strong scaling speedup per timestep of up to 40x in three-dimensional phase space (one position, two velocity coordinates) and 54x in four dimensional phase space (two position, two velocity coordinates) and a 341x increase in simulation throughput of the GPU accelerated code over the existing CPU code. The GPU code is also able to weak scale up to 256 compute nodes and 1024 GPUs. We then demonstrate how the improved compute performance can be used to explore configurations which were previously computationally infeasible such as resolving fine-scale distribution function filamentation and multi-species dynamics with realistic electron-proton mass ratios.