惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

云风的 BLOG
云风的 BLOG
The GitHub Blog
The GitHub Blog
A
About on SuperTechFans
P
Proofpoint News Feed
G
Google Developers Blog
Stack Overflow Blog
Stack Overflow Blog
IT之家
IT之家
Microsoft Security Blog
Microsoft Security Blog
F
Fortinet All Blogs
人人都是产品经理
人人都是产品经理
博客园 - 叶小钗
C
Check Point Blog
Microsoft Azure Blog
Microsoft Azure Blog
aimingoo的专栏
aimingoo的专栏
月光博客
月光博客
美团技术团队
D
Docker
博客园 - Franky
Y
Y Combinator Blog
大猫的无限游戏
大猫的无限游戏
Cyber Security Advisories - MS-ISAC
Cyber Security Advisories - MS-ISAC
博客园 - 【当耐特】
罗磊的独立博客
奇客Solidot–传递最新科技情报
奇客Solidot–传递最新科技情报

stat updates on arXiv.org

Simultaneous Monitoring of Shape and Surface Color via 4D Point Clouds: A Registration-free Approach A Refined Generalization Analysis for Extreme Multi-class Supervised Contrastive Representation Learning Ensemble Distributionally Robust Bayesian Optimisation The Proxy Presumption: From Semantic Embeddings to Valid Social Measures Modulated learning for private and distributed regression with just a single sample per client device Query-efficient model evaluation using cached responses Functional-prior-based approaches to Bayesian PDE-constrained inversion using physics-informed neural networks Optimal Experiments for Partial Causal Effect Identification Order-Agnostic Autoregressive Modelling with Missing Data Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes Tuning Derivatives for Causal Fairness in Machine Learning Spherical Flows for Sampling Categorical Data Bayesian Rain Field Reconstruction using Commercial Microwave Links and Diffusion Model Priors GRALIS: A Unified Canonical Framework for Linear Attribution Methods via Riesz Representation Sharp Capacity Thresholds in Linear Associative Memory: From Winner-Take-All to Listwise Retrieval Unified Framework of Distributional Regret in Multi-Armed Bandits and Reinforcement Learning Jacobian-Velocity Bounds for Deployment Risk Under Covariate Drift Self-Attention as Transport: Limits of Symmetric Spectral Diagnostics Perturbation is All You Need for Extrapolating Language Models Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization Realizable Bayes-Consistency for General Metric Losses Graph Convolutional Support Vector Regression for Robust Spatiotemporal Forecasting of Urban Air Pollution Segmenting Human-LLM Co-authored Text via Change Point Detection Stochastic Schrödinger Diffusion Models for Pure-State Ensemble Generation Understanding Self-Supervised Learning via Latent Distribution Matching The Geometric Mechanics of Contrastive Representation Learning: Alignment Potentials, Entropic Dispersion, and Cross-modal Divergence Imbalanced Classification under Capacity Constraints On the Spectral Structure and Objective Equivalence of Orthogonal Multilabel Fisher Discriminants Partially Observed Structural Causal Models First-Order Efficiency for Probabilistic Value Estimation via A Statistical Viewpoint
Learning large softmax mixtures with warm start EM
Xin Bing, Florentina Bunea, Jonathan Niles-Weed, Marten Wegkamp · 2024-09-16 · via stat updates on arXiv.org

Softmax mixture models (SMMs) are discrete $K$-mixtures introduced to model the probability of choosing an attribute $x_j \in \RR^L$ from $p$ candidates, in heterogeneous populations. They have been known as mixed multinomial logits in the econometrics literature, and are gaining traction in the LLM literature, where single softmax models are routinely used in the final layer of a neural network. This paper provides a comprehensive analysis of the EM algorithm for SMMs in high dimensions. Its population-level theoretical analysis forms the basis for proving (i) local identifiability, in SSMs with generic features and, further, via a stochastic argument, (ii) full identifiability in SSMs with random features, when $p$ is large enough. These are the first results in this direction for SSMs with $L > 1$. The population-level EM analysis characterizes the initialization radius for algorithmic convergence. This also guides the construction of warm starts of the sample level EM. Under suitable initialization, the EM algorithm is shown to recover the mixture atoms of the SSM at near-parametric rate. We provide two main directions for warm start construction, both based on a new method for estimating the moments of the mixing measure underlying an SSM with random design. First, we construct a method of moments (MoM) estimator of the mixture parameters, and provide its first theoretical analysis. While MoM can enjoy parametric rates of convergence, and thus can serve as a warm-start, the estimator's quality degrades exponentially in $K$. Our recommendation, when $K$ is not small, is to run the EM algorithm several times with random initializations. We again make use of the novel latent moments estimation method to estimate the $K$-dimensional subspace of the mixture atoms. Sampling from this subspace reduces substantially the number of required draws.