By integrating edge computing with parallel computing, distributed edge computing (DEC) makes use of distributed devices in edge networks to perform computing in parallel, which can substantially reduce service delays. In this paper, we explore DEC that exploits distributed edge devices connected by a wireless network to perform a computation task offloaded from an end device. In particular, we study the fundamental problem of minimizing the delay of executing a distributed algorithm of the computation task. We first establish some structural properties of the optimal communication scheduling policy. Then, given these properties, we characterize the optimal computation allocation policy, which can be found by an efficient algorithm. Next, based on the optimal computation allocation, we characterize the optimal scheduling order of communications for some special cases, and develop an efficient algorithm with a finite approximation ratio to find it for the general case. Last, based on the optimal computation allocation and communication scheduling, we further show that the optimal selection of devices can be found efficiently for some special cases. Our results provide some useful insights for the optimal computation-communication co-design. We evaluate the performance of the theoretical findings using simulations.