惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

Google DeepMind News
Google DeepMind News
C
Check Point Blog
GbyAI
GbyAI
Jina AI
Jina AI
Cyber Security Advisories - MS-ISAC
Cyber Security Advisories - MS-ISAC
博客园 - 司徒正美
Hacker News - Newest:
Hacker News - Newest: "LLM"
V2EX - 技术
V2EX - 技术
cs.CV updates on arXiv.org
cs.CV updates on arXiv.org
F
Full Disclosure
S
Secure Thoughts
WordPress大学
WordPress大学
博客园 - Franky
N
News and Events Feed by Topic
雷峰网
雷峰网
www.infosecurity-magazine.com
www.infosecurity-magazine.com
H
Hacker News: Front Page
N
Netflix TechBlog - Medium
Forbes - Security
Forbes - Security
TaoSecurity Blog
TaoSecurity Blog
C
CERT Recently Published Vulnerability Notes
CTFtime.org: upcoming CTF events
CTFtime.org: upcoming CTF events
cs.AI updates on arXiv.org
cs.AI updates on arXiv.org
Vercel News
Vercel News
T
The Blog of Author Tim Ferriss
MyScale Blog
MyScale Blog
Security Latest
Security Latest
Attack and Defense Labs
Attack and Defense Labs
The Register - Security
The Register - Security
Spread Privacy
Spread Privacy
H
Help Net Security
宝玉的分享
宝玉的分享
L
LangChain Blog
MongoDB | Blog
MongoDB | Blog
I
Intezer
Schneier on Security
Schneier on Security
Latest news
Latest news
Y
Y Combinator Blog
Recent Commits to openclaw:main
Recent Commits to openclaw:main
I
InfoQ
A
About on SuperTechFans
T
Tor Project blog
Microsoft Security Blog
Microsoft Security Blog
M
MIT News - Artificial intelligence
Google DeepMind News
Google DeepMind News
Microsoft Azure Blog
Microsoft Azure Blog
T
Threat Research - Cisco Blogs
S
Schneier on Security
Cisco Talos Blog
Cisco Talos Blog
H
Heimdal Security Blog

cs updates on arXiv.org

Beyond Binary Edits Robust Multimodal Knowledge Editing with Adversarial Subspace Alignment Agentic Proving for Program Verification MemAudit: Post-hoc Auditing of Poisoned Agent Memory via Causal Attribution and Structural Anomaly Detection OpenSkillEval: Automatically Auditing the Open Skill Ecosystem for LLM Agents One Policy, Infinite NPCs: Persona-Traceable Shared RL Policies for Scalable Game Agents How Human-Like Are Large Language Models? A Register-Aware Linguistic Evaluation Framework Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems Structure-Guided Entity Resolution: Fine-Tuning LLMs for Robust Name Matching in Complex Linguistic Contexts Solving the Aircraft Disassembly Scheduling Problem Co-ReAct: Rubrics as Step-Level Collaborators for ReAct Agents CP or DP? Why Not Both: A Case Study in the Partial Shop Scheduling Problem Asking For An Old Friend: Diagnosing and Mitigating Temporal Failure Modes in LLM-based Statutory Question Answering EDGE-OPD: Internalizing Privileged Context with Evidence Guided On-Policy Distillation ARES: Automated Rubric Synthesis for Scalable LLM Reinforcement Learning SSDAU: Structured Semantic Data Augmentation for Joint Entity and Relation Extraction Naturalistic measure of social norms alignment Articulatory strategy as a source of variation in acoustic vowel dynamics When Planning Fails Despite Correct Execution: On Epistemic Calibration for LLM-Based Multi-Agent Systems EquiSumm : A Gender Bias-Aware Framework for Inclusive Tweet Summarization Metacognition as Reward: Reinforcing LLM Reasoning via Knowledge and Regulation Signals From Correctness to Preference: A Framework for Personalized Agentic Reinforcement Learning Cultural Adaptation in Large Language Models for Political Discourse Emotion Recognition in Sign Language Conversation ClimateChat-300K: A Multi-Modal Facebook Dataset for Understanding Diverse Perspectives in Climate Communication AraHopeCorpus: Annotation Guidelines and Dataset for Hope Speech in Arabic Social Media Crisis Discourse Human-in-the-Loop Multi-Agent Ventilator Decision Support with Contextual Bandit Preference Learning Convergence Without Understanding: When Language Models Agree on Representations but Disagree on Reasoning DART: Semantic Recoverability for Structured Tool Agents Ontological Knowledge Blocks: Executable Compliance and Profile-Based Validation for Trustworthy AI Systems Parallel Context Compaction for Long-Horizon LLM Agent Serving When Is Next-Token Prediction Useful? Marginalization, Ergodicity, Mixture Identifiability, Local Sufficiency, RAG, Tools, and Programming Design and Report Benchmarks for Knowledge Work GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models Foundation Protocol: A Coordination Layer for Agentic Society AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discovery Hidden Human-Like Nature of Machine-Generated Texts: Theory and Detection Enhancement Self-Improving In-Context Learning Redrawing the AI Map: A Theory of Accountability Boundaries in Agentic Ecosystems Positional Failures in Long-Context LLMs: A Blind Spot in Reasoning Benchmarks Fast-dDrive: Efficient Block-Diffusion VLM for Autonomous Driving Same Model, Different Weakness: How Language and Modality Reshape the Jailbreak Attack Surface in Frontier MLLMs When Symptoms Are Not Enough: Evidence-Weighting Patterns in Large Language Model Psychiatric Screening As X, Do Y: How Persona and Task Combine in Instruction-Tuned LLMs CoReVAD: A Contextual Reasoning Framework for Training-Free Video Anomaly Detection Inconsistency-aware Multimodal Schrödinger Bridge for Deepfake Localization Inductive Deductive Synthesis: Enabling AI to Generate Formally Verified Systems A Fine-Tuned BERT Classifier for Personal-Letter Titles in Late-Ming and Early-Qing Collected Works A Comparative Evaluation of Structural Topic Models and BERTopic for Short, Open-Ended Survey Responses PathCal: State-Aware Reflection-Marker Calibration for Efficient Reasoning The Efficiency Frontier: A Unified Framework for Cost-Performance Optimization in LLM Context Management Flow Mismatching: Unsupervised Anomaly Detection via Velocity Discrepancies in Flow Matching Models DFKI-MLT at SemEval-2026 TASK 7: Steering Multilingual Models Towards Cultural Knowledge RoboSurg-VQA: A Multimodal Benchmark for Surgical Segmentation-Aware Visual Question Answering What Training Data Teaches RL Memory Agents: An Empirical Study of Curriculum Effects in Memory-Augmented QA Dithering Defense: Adversarial Robustness of Vision Foundation Models via Multi-Level Floyd-Steinberg Dithering Millimeter-wave Imaging for Anthropometric Body Measurement Model Collapse as Cultural Evolution DreamerNLplus: Interpretable Modeling of Mental Health Dynamics from Social Media Timelines using Hybrid Rule-Based and RAG Methods The TIME Machine: On The Power of Motion for Efficient Perception HawkesLLM: Semantic Uncertainty Propagation in Agentic Text Simulation Do Language Models Know What Not to Say? Causal Evidence for Statistical Preemption in LLMs Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selection Sparse Autoencoders Map Brain-LLM Alignment onto Cortical Semantic Topography Brain-LLM Alignment Tracks Training Data, Not Typology The Deterministic Horizon: Impossibility Results as Design Specifications for Trustworthy AI Systems Scene Reconstruction as Mapping Priors for 3D Detection CoMoGen: COntrollable MOtion Dynamics and Interactions with Mask-Guided Video GENeration A Proactive Multi-Agent Dialogue Framework for Assessing Social Language Disorder Traits in Autism Memorization Dynamics of Fill-in-the-Middle Pretraining A Reproducible Universal Dependencies-Style Pipeline for Katharevousa Greek Parliamentary Text When AI Takes Sides on Questions of Faith: Persistent Asymmetries in AI-Mediated Faith Guidance Can AI Guess What You Know? Performance Comparison of Large Language Models for Human Domain Knowledge Estimation From Communication Logs Graph Alignment Topology as an Inductive Bias for Grounding Detection GazeBehavior Annotation Toolkit (GBAT): AI-powered toolkit for automatic annotation of egocentric eye-tracking and video data of child-caregiver interaction Improved Vision-to-Chart Buoy Association with Learned World-to-Image Projection Learnability-Informed Fine-Tuning of Diffusion Language Models RAS: Reflection-Augmented Scaling with In-Context Learning for Executable Cypher Query Generation VideoOdyssey: A Benchmark for Ultra-Long-Context and Omni-Modal Video Understanding EVE-Agent: Evidence-Verifiable Self-Evolving Agents Suicide Risk Assessment from AI-powered Video Surveillance: An Interpretable Framework for Prevention in Metro Stations Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision? Mediative Fuzzy Logic: From Type-1 Foundations to Type-2, Type-3 and Quantum Extensions ImProver 2: Iteratively Self-Improving LMs for Neurosymbolic Proof Optimization Energy per Successful Goal: Goal-Level Energy Accounting for Agentic AI Systems GEM-4D: Geometry-Enhanced Video World Models for Robot Manipulation How Far Will They Go? Red-Teaming Online Influence with Large Language Models SciAtlas: A Large-Scale Knowledge Graph for Automated Scientific Research RMA: an Agentic System for Research-Level Mathematical Problems NeuroNL2LTL: A Neurosymbolic Framework for Natural Language Translation of Linear Temporal Logic BOHM: Zero-Cost Hierarchical Attribution for Compound AI Systems GAGPO: Generalized Advantage Grouped Policy Optimization Knowledge Distillation for Low-Resource Open-source Text-to-SQL Model Query-Adaptive Semantic Chunking for Retrieval-Augmented Generation: A Dynamic Strategy with Contextual Window Expansion A Survey of Text and Speech Resources for Hausa and Fongbe: Availability, Quality, and Gaps for NLP Development Evaluating Large Language Models in a Complex Hidden Role Game AV-Master: Dual-Path Comprehensive Perception Makes Better Audio-Visual Question Answering Memory-SAM: Human-Prompt-Free Tongue Segmentation via Retrieval-to-Prompt DIVER: Reinforced Diffusion Breaks Imitation Bottlenecks in End-to-End Autonomous Driving Transformer-Empowered Actor-Critic Reinforcement Learning for Sequence-Aware Service Function Chain Partitioning VerteNet -- A Multi-Context Hybrid CNN Transformer for Accurate Vertebral Landmark Localization in Lateral Spine DXA Images
The African Language Tax: Quantifying the Cost, Latency, and Context Penalty of Tokenizing African Languages in Frontier LLMs
[Submitted on 23 Jun 2026] · 2026-06-24 · via cs updates on arXiv.org

View PDF HTML (experimental)

Abstract:Commercial large language models bill, scale latency, and budget context per token. Yet tokenizers assign more subword tokens to the same meaning in some languages than in others, so speakers of languages with high token-fertility pay a structural penalty before a model is ever invoked. This penalty is documented for multilingual settings in general, but it has not been measured systematically for African languages at the level of enterprise deployment economics and cognitive context capacity. We measure it across 20 African languages spanning five language families and three scripts (Latin, Ge'ez/Ethiopic, N'Ko; 19 appear in the primary FLORES-200+ corpus, with Nigerian Pidgin measured via MAFAND-MT only), using parallel corpora so that the language effect is isolated from content. Across 11 frontier and open tokenizers on FLORES-200+, every African language carries a tokenization premium above English (median 1.88x on GPT-5 / o200k_base, up to 8.92x for N'Ko); the penalty is largest for Ethiopic and N'Ko scripts (reaching 7-9x) and is near-invariant across corpora (FLORES vs SIB-200 Pearson r = 0.9998). Translated into deployment terms, this results in up to 8.9x inference cost and an equivalent generation-latency multiplier (N'Ko vs English on GPT-5; 7.4x for Amharic), and as little as 11% of English's effective context window. The best currently available tokenizer for African languages, Gemma 4, reduces the mean premium from 3.31x (cl100k_base) to 2.38x, but no tokenizer eliminates the penalty. We release an open measurement tool (afri-fertility), a public leaderboard, a results dataset, and mitigation guidance for African builders. The penalty falls hardest on the languages whose speakers can least afford it, a digital divide encoded directly into the subword vocabulary.

Submission history

From: Olaoye Anthony Somide [view email]
[v1] Tue, 23 Jun 2026 11:47:03 UTC (731 KB)