Unmanned aerial vehicles (UAVs) often collaborate by collecting and offloading sensing streams to an edge server, where a deep neural network (DNN) model performs cross-stream alignment, fusion, and inference. However, the coupling between wireless offloading and DNN execution makes end-to-end latency minimization challenging. To address this issue, this paper investigates efficient edge inference in multi-UAV networks. Specifically, a multi-UAV collaborative edge inference model is first established, in which UAV sensing streams are processed by a multi-branch DNN on a multi-core accelerator. Based on this model, an end-to-end latency minimization problem with a synchronization penalty is formulated. A genetic algorithm (GA)-based full joint scheduler, termed \texttt{GA-Joint}, is then developed to obtain high-quality scheduling solutions. To reduce the search complexity, two lightweight variants, termed \texttt{GA-DAG} and \texttt{GA-DACS}, are further proposed. Simulation results demonstrate that the proposed GA-based scheduling algorithms achieve lower end-to-end latency than \texttt{Decoupled-Greedy} and \texttt{Joint-Greedy}, which represent decoupled and joint greedy scheduling schemes, respectively, in most cases. Furthermore, \texttt{GA-DACS} achieves performance close to that of \texttt{GA-Joint} in many cases and even delivers slightly lower latency in certain scenarios.