惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

V
V2EX
aimingoo的专栏
aimingoo的专栏
S
SegmentFault 最新的问题
博客园_首页
OSCHINA 社区最新新闻
OSCHINA 社区最新新闻
IT之家
IT之家
博客园 - 【当耐特】
月光博客
月光博客
C
Check Point Blog
T
The Blog of Author Tim Ferriss
罗磊的独立博客
博客园 - Franky
MongoDB | Blog
MongoDB | Blog
H
Help Net Security
Microsoft Security Blog
Microsoft Security Blog
B
Blog
阮一峰的网络日志
阮一峰的网络日志
腾讯CDC
美团技术团队
N
Netflix TechBlog - Medium
Stack Overflow Blog
Stack Overflow Blog
Y
Y Combinator Blog
L
LangChain Blog
The Cloudflare Blog

cs.DC updates on arXiv.org

DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference Progressive Semantic Communication for Efficient Edge-Cloud Vision-Language Models Efficient, VRAM-Constrained xLM Inference on Clients Folding Tensor and Sequence Parallelism for Memory-Efficient Transformer Training & Inference DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving RaMP: Runtime-Aware Megakernel Polymorphism for Mixture-of-Experts Spark Policy Toolkit: Semantic Contracts and Scalable Execution for Policy Learning in Spark Internet of Everything in the 6G Era: Paradigms, Enablers, Potentials and Future Directions PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference A Survey on Split Learning for LLM Fine-Tuning: Models, Systems, and Privacy Optimizations ITAS: A Multi-Agent Architecture for LLM-Based Intelligent Tutoring Latency and Cost of Multi-Agent Intelligent Tutoring at Scale TACO: Efficient Communication Compression of Intermediate Tensors for Scalable Tensor-Parallel LLM Training FreeScale: Distributed Training for Sequence Recommendation Models with Minimal Scaling Cost CommFuse: Hiding Tail Latency via Communication Decomposition and Fusion for Distributed LLM Training A Taxonomy and Resolution Strategy for Client-Level Disagreements in Federated Learning Usable Agent Discovery for Decentralized AI Systems Cloud to Edge: Benchmarking LLM Inference On Hardware-Accelerated Single-Board Computers Data-Free Contribution Estimation in Federated Learning using Gradient von Neumann Entropy Shard the Gradient, Scale the Model: Serverless Federated Aggregation via Gradient Partitioning Promoting Simple Agents: Ensemble Methods for Event-Log Prediction GraphLeap: Decoupling Graph Construction and Convolution for Vision GNN Acceleration on FPGA AGNT2: Autonomous Agent Economies on Interaction-Optimized Layer 2 Infrastructure FedSIR: Spectral Client Identification and Relabeling for Federated Learning with Noisy Labels Stream-CQSA: Avoiding Out-of-Memory in Attention Computation via Flexible Workload Scheduling A Delta-Aware Orchestration Framework for Scalable Multi-Agent Edge Computing Federated Learning over Blockchain-Enabled Cloud Infrastructure Optimal Routing for Federated Learning over Dynamic Satellite Networks: Tractable or Not? Sherpa.ai Privacy-Preserving Multi-Party Entity Alignment without Intersection Disclosure for Noisy Identifiers
Scalable and Efficient Construction of Suffix Array with ...
Hsiang-Huang Wu, Chien-Min Wang, Hsuan-Chi Kuo, Wei-Chun Chung, · 2017-05-13 · via cs.DC updates on arXiv.org

Suffix Array (SA) is a cardinal data structure in many pattern matching applications, including data compression, plagiarism detection and sequence alignment. However, as the volumes of data increase abruptly, the construction of SA is not amenable to the current large-scale data processing frameworks anymore due to its intrinsic proliferation of suffixes during the construction. That is, ameliorating the performance by just adding the resources to the frameworks becomes less cost- effective, even having the severe diminishing returns. At issue now is whether we can permit SA construction to be more scalable and efficient for the everlasting accretion of data by creating a radical shift in perspective. Regarding TeraSort [1] as our baseline, we first demonstrate the fragile scalability of TeraSort and investigate what causes it through the experiments on the sequence alignment of a grouper (i.e., the SA construc- tion used in bioinformatics). As such, we propose a scheme that amalgamates the distributed key-value store system into MapReduce to leverage the in-memory queries about suffixes. Rather than handling the communication of suffixes, MapReduce is in charge of the communication of their indexes, which means better capacity for more data. It significantly abates the required disk space for constructing SA and better utilizes the memory, which in turn improves the scalability radically. We also examine the efficiency of our scheme in terms of memory and show it outperforms TeraSort. At last, our scheme can complete the pair- end sequencing and alignment with two input files without any degradation on scalability, and can accommodate the suffixes of nearly 6.7 TB in a small cluster composed of 16 nodes and Gigabit Ethernet without any compression.