

















Abstract:Distributed scientific workflows are increasingly executed across heterogeneous and geo-distributed computing environments, where centralized workload orchestration becomes a scalability and resilience bottleneck. This paper presents SWARM+, a decentralized workload management system that coordinates workload placement through hierarchical multi-agent consensus, reducing coordination overhead and dramatically improving scalability, while tolerating failures and dynamic membership changes. SWARM+ enables data-aware scheduling policies that incorporate resource availability, data transfer node (DTN) connectivity, and data locality into workload placement decisions. We evaluate SWARM+ on the distributed FABRIC testbed using heterogeneous scientific workloads derived from production workflow traces obtained from the Pegasus Workflow Management System (WMS). Experimental results show that SWARM+ scales coordination to 990 distributed agents with approximately 1\,s per-job selection time at 110 agents. SWARM+ demonstrates balanced workload distribution, maintains over 97% job completion under distributed failures with graceful degradation (mean ~95% job completion) during correlated site outages, tolerates coordinator agent failures gracefully, improves schedule quality by employing data-aware policies, and reduces both selection time and scheduling latency by 97-98% when compared to the prior SWARM system.
From: Komal Thareja [view email]
[v1]
Thu, 19 Mar 2026 19:51:02 UTC (1,300 KB)
[v2]
Tue, 2 Jun 2026 15:09:01 UTC (458 KB)
[v3]
Mon, 8 Jun 2026 17:43:01 UTC (398 KB)
[v4]
Tue, 14 Jul 2026 17:51:22 UTC (398 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。