神经网络的“彩票理论”是人工智能领域的深度神经网络的一篇很重要的实证的理论论文，也就是使用实证的方法给出对深度神经网络的理论猜测，但是该论文最大的问题就是只能在理论上给出解释，并不难被应用于实际的商业环境下，无法真正的落地，因为该论文给出的稀疏网络是不能在现有的神经网络芯片上进行加速运算的，而该理论最初的提出者也明确表示不再推进该项工作了。

原讨论地址：

https://news.ycombinator.com/item?id=46470513

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018) (arxiv.org)
121 points by felineflock 3 months ago | hide | past | favorite | 27 comments

	
laughingcurve 3 months ago | next [–]

Article from 2018/19 and this hypothesis remains just that afaik with plenty of evidence going against it

	
swyx 3 months ago | parent | next [–]

i intereviewed Jon (lead author on this paper) and yeah he pretty much disowns it now https://www.latent.space/p/mosaic-mpt-7b

	
gwern 3 months ago | root | parent | next [–]

Could you explain why you think that? I'm looking at the lottery ticket section and it seems like he doesn't disown it; the reason he gives, via Abhinav, for not pursuing it at his commercial job is just that that kind of sparsity is not hardware friendly (except with Cerebras). "It doesn't provide a speedup for normal commercial workloads on normal commercial GPUs and that's why I'm not following it up at my commercial job and don't want to talk about it" seems pretty far from "disowning the lottery ticket hypothesis [as wrong or false]".

	
oofbey 3 months ago | root | parent | next [–]

I think that was pretty clear even when this paper came out - even if you could find these sub networks they wouldn’t be faster on real hardware. Never thought much of this paper, but it sure did get a lot of people excited.

	
sailingparrot 3 months ago | root | parent | next [–]

It was exciting because of what it means regarding how a model learns, regardless on whether or not its commercially applicable.

	
gwern 3 months ago | root | parent | prev | next [–]

(Cerebras is real hardware.)

	
oofbey 3 months ago | root | parent | next [–]

It is real in that it exists. It is not real in the sense that almost nobody has access to them. Unless you work at one of the handful of organizations with their hardware, it’s not a practical reality.

	
aaronblohowiak 3 months ago | root | parent | next [–]

how long will that be the case?

	
oofbey 3 months ago | root | parent | next [–]

They have a strange business model. Their chips are massive. So they necessarily only sell them to large customers. Also because of the way they’re built (entire wafer is a single chip) no two chips will be the same. Normally imperfections in the manufacturing result in some parts of the wafer being rejected and other binned as fast or slow chips. If you use the whole wafer you get what you get. So it’s necessarily a strange platform to work with - every device is slightly different.

	
IshKebab 3 months ago | root | parent | prev | next [–]

At least for the foreseeable future (next 50 years say).

	
laughingcurve 3 months ago | root | parent | prev | next [–]

i saw how it nerdsniped an extremely capable faculty member

	
swyx 3 months ago | root | parent | prev | next [–]

he pretty much always says it offline haha but i maay have mixed it up with the subsequent convo we had at neurips https://www.latent.space/p/neurips-2023-startups

	
laughingcurve 3 months ago | root | parent | prev | next [–]

cool beans, thanks for this -- I think it's easier to hear it directly from the authors. I was hesitant to start researchposting and come off like a dick.
also; note to self: If I publish and disown my papers, shawn will interview me :)


	
yorwba 3 months ago | parent | prev | next [–]

What evidence against it do you have in mind? I think it's a result of little practical relevance without a way to identify winning tickets that doesn't require buying lots of tickets until you hit the jackpot (i.e. training a large, dense model to completion) but that doesn't make the observation itself incorrect.

	
kingstnap 3 months ago | root | parent | next [–]

The observation itself is also partially incorrect. This is a video I watched a few months ago that went further into the whole how do you deal with subnetworks thing.
https://youtu.be/WW1ksk-O5c0?list=PLCq6a7gpFdPgldPSBWqd2THZh... (timestamped)

At the timestamp they discuss how actually the original ICLR results only worked on these extremely tiny models and larger ones didn't work. The adaptation you need to sort of fix it is to train densely first for a few epochs, only then you can start increasing sparsity.


	
paulsutter 3 months ago | root | parent | next [–]

Watched the video - thanks
Ioannu is saying the paper's idea for training a dense network doesn't work in non-toy networks (the paper's method for selecting promising weights early doesn't improve the network)

BUT the term "lottery ticket" refers to the true observation that a small subset of weights drive functionality (see all pruning papers). It's great terminology because they truly are coincidences based on random numbers.

All that's been disproven is that paper's specific method to create a dense network based on this observation


	
rob_c 3 months ago | prev | next [–]

This is basically just a rehash of "trained" DNN are a function which is strongly dependent on the initialization parameters. (Easily provable)
It would be awesome to have a way of finding them in advance but this is also just a case of avoid pure DNNs due to their strong reliance on initialization parameters.

Looking at transformers by comparison you see a much much weaker dependence of the model on the input initial parameters. Does this mean the model is better or worse at learning or just more stable?


	
snaking0776 3 months ago | parent | next [–]

This is an interesting insight I hadn’t thought much about before. Reminds me a bit of some of the mechanistic interpretability work that looked at branch specialization in CNNs and found that architectures which had built in branches tended to have those branches specialize in a way that was consistent across multiple training runs [1]. Maybe the multi-headed and branching nature of transformers adds and inductive bias that is useful for stable training over larger scales.
[1] https://distill.pub/2020/circuits/branch-specialization/


	
observationist 3 months ago | prev | next [–]

Neural networks are effectively gauge invariant, and you have a huge space of valid isomorphisms as far as possible "valid" layer orderings go, and if your network is overparameterized, the space of "good enough" approximations gets correspondingly larger. The good enough sets are a sort of fuzzy gauge quotient approximating some "ideal" function per layer or cluster or block (depending on your optimizer and architecture.)
https://arxiv.org/html/2506.13018v2 - Here's an interesting paper that can help inform how you might look at networks, especially in the context of lottery tickets, gauge quotients, permutations, and what gradient descent looks like in practice.

Kolmogorov Arnold Networks are better about exposing gauge symmetry and operating in that space, but aren't optimized for the hardware we have - mechinterp and other reasons might inspire new hardware, though. If you know what your layer function should look like, if it were ordered such that it resembled a smooth spline, you could initialize and freeze the weights of that layer, and force the rest of the network to learn within the context of your chosen ordering.

The number of "valid" configurations for a layer is large, especially if you have more neurons in the layer than you need, and the number of subsequent layer configurations is much larger than you'd think. The lottery ticket hypothesis is just circling that phenomenon without formalizing it - some surprisingly large percentage of possible configurations will approximate the function you want a network to learn. It doesn't necessarily gain you advantages in achieving the last 10% , and there could be counterproductive configurations that collapse before reaching an optimal configuration.

There are probably optimizer strategies that can exploit initializations of certain types, for different classes of activation functions, and achieve better performance for architectures - and all of those things are probably open to formalized methods based on existing number theory around gauge invariant systems and gauge quotients, with different layer configurations existing as points in gauge orbits in hyperdimensional spaces.

It'd be really cool if you could throw twice as many neurons as you need into a model, randomly initialize a bunch of times until you get a winning ticket, then distill the remainder down to your intended parameter count, and train from there as normal.

It's more complex with architectures like transformers, but you're not dealing with a combinatorial explosion with the LTH - more like a little combinatorial flash flood, and if you engineer around it, it can actually be exploited.


	
pizza 3 months ago | parent | next [–]

Yes to this. Furthermore:
- you can solve neural networks in analytic form with a hodge star approach* [0]

- if you use a picture to set your initial weights for your nn, you can see visually how close or far your choice of optimizer is actually moving the weights - eg non-dualized optimizers look like they barely change things whereas dualized Muon changes the weights much more to the point you cannot recognize the originals [1]

*unfortunately, this is exponential in memory

[0] M. Pilanci — From Complexity to Clarity: Analytical Expressions of Deep Neural Network Weights via Clifford's Geometric Algebra and Convexity https://arxiv.org/abs/2309.16512

[1] https://docs.modula.systems/examples/weight-erasure/


	
eru 3 months ago | root | parent | next [–]

Thanks for the explanations and the great links!

	
srean 3 months ago | parent | prev | next [–]

Wouldn't such local invariance tie in with flatness or shallowness of the minima ?
This would tie in with the observation that flat/shallow minimas are easier to find with stochastic gradient descent and such weights generalise better.


	
sbinnee 3 months ago | prev | next [–]

I was referring to this paper a lot when it was hyped, when people cared about architectural decisions of neural networks. It was also the year I started studying neural networks.
I think the idea still holds. Although the interest has been shifted towards test-time scaling and thinking, researcher still care about architectures like nemotron 3, recently published.

Can anyone give more updates on this direction of research, more recent papers?


	
choult 3 months ago | prev | next [–]

_Fewer_

	
eru 3 months ago | parent | next [–]

https://en.wikipedia.org/wiki/Fewer_versus_less
Compare also http://fine.me.uk/Emonds/wholetext.xml


	
tomhow 3 months ago | parent | prev | next [–]

Indeed, the original title didn't make that mistake, so we've restored the original title as per the guidelines.

	
mceachen 3 months ago | prev [–]

@dang please retitle with (2018)

以下是豆包AI大模型的翻译：

假设你尝试构造一张失效彩票子网络：只保留不属于获胜彩票的那些权重，并从这些权重开始重新训练。
这样的模型会完全无法学习该任务吗？还是说，在这些权重之中，可能还藏着另一张原本未被发现的获胜彩票子网络？
术语说明（贴合论文语境）
winning ticket：获胜彩票（彩票假设中可独立训练、性能接近完整网络的稀疏子网络）
losing lottery ticket：失效彩票（无法有效学习的子网络）
retraining from those：基于这些权重重新初始化训练






The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018) (arxiv.org) 121 points by felineflock 3 months ago | hide | past | favorite | 27 comments laughingcurve 3 months ago | next [–] Article from 2018/19 and this hypothesis remains just that afaik with plenty of evidence going against it swyx 3 months ago | parent | next [–] i intereviewed Jon (lead author on this paper) and yeah he pretty much disowns it now https://www.latent.space/p/mosaic-mpt-7b gwern 3 months ago | root | parent | next [–] Could you explain why you think that? I'm looking at the lottery ticket section and it seems like he doesn't disown it; the reason he gives, via Abhinav, for not pursuing it at his commercial job is just that that kind of sparsity is not hardware friendly (except with Cerebras). "It doesn't provide a speedup for normal commercial workloads on normal commercial GPUs and that's why I'm not following it up at my commercial job and don't want to talk about it" seems pretty far from "disowning the lottery ticket hypothesis [as wrong or false]". oofbey 3 months ago | root | parent | next [–] I think that was pretty clear even when this paper came out - even if you could find these sub networks they wouldn’t be faster on real hardware. Never thought much of this paper, but it sure did get a lot of people excited. sailingparrot 3 months ago | root | parent | next [–] It was exciting because of what it means regarding how a model learns, regardless on whether or not its commercially applicable. gwern 3 months ago | root | parent | prev | next [–] (Cerebras is real hardware.) oofbey 3 months ago | root | parent | next [–] It is real in that it exists. It is not real in the sense that almost nobody has access to them. Unless you work at one of the handful of organizations with their hardware, it’s not a practical reality. aaronblohowiak 3 months ago | root | parent | next [–] how long will that be the case? oofbey 3 months ago | root | parent | next [–] They have a strange business model. Their chips are massive. So they necessarily only sell them to large customers. Also because of the way they’re built (entire wafer is a single chip) no two chips will be the same. Normally imperfections in the manufacturing result in some parts of the wafer being rejected and other binned as fast or slow chips. If you use the whole wafer you get what you get. So it’s necessarily a strange platform to work with - every device is slightly different. IshKebab 3 months ago | root | parent | prev | next [–] At least for the foreseeable future (next 50 years say). laughingcurve 3 months ago | root | parent | prev | next [–] i saw how it nerdsniped an extremely capable faculty member swyx 3 months ago | root | parent | prev | next [–] he pretty much always says it offline haha but i maay have mixed it up with the subsequent convo we had at neurips https://www.latent.space/p/neurips-2023-startups laughingcurve 3 months ago | root | parent | prev | next [–] cool beans, thanks for this -- I think it's easier to hear it directly from the authors. I was hesitant to start researchposting and come off like a dick. also; note to self: If I publish and disown my papers, shawn will interview me :) yorwba 3 months ago | parent | prev | next [–] What evidence against it do you have in mind? I think it's a result of little practical relevance without a way to identify winning tickets that doesn't require buying lots of tickets until you hit the jackpot (i.e. training a large, dense model to completion) but that doesn't make the observation itself incorrect. kingstnap 3 months ago | root | parent | next [–] The observation itself is also partially incorrect. This is a video I watched a few months ago that went further into the whole how do you deal with subnetworks thing. https://youtu.be/WW1ksk-O5c0?list=PLCq6a7gpFdPgldPSBWqd2THZh... (timestamped) At the timestamp they discuss how actually the original ICLR results only worked on these extremely tiny models and larger ones didn't work. The adaptation you need to sort of fix it is to train densely first for a few epochs, only then you can start increasing sparsity. paulsutter 3 months ago | root | parent | next [–] Watched the video - thanks Ioannu is saying the paper's idea for training a dense network doesn't work in non-toy networks (the paper's method for selecting promising weights early doesn't improve the network) BUT the term "lottery ticket" refers to the true observation that a small subset of weights drive functionality (see all pruning papers). It's great terminology because they truly are coincidences based on random numbers. All that's been disproven is that paper's specific method to create a dense network based on this observation rob_c 3 months ago | prev | next [–] This is basically just a rehash of "trained" DNN are a function which is strongly dependent on the initialization parameters. (Easily provable) It would be awesome to have a way of finding them in advance but this is also just a case of avoid pure DNNs due to their strong reliance on initialization parameters. Looking at transformers by comparison you see a much much weaker dependence of the model on the input initial parameters. Does this mean the model is better or worse at learning or just more stable? snaking0776 3 months ago | parent | next [–] This is an interesting insight I hadn’t thought much about before. Reminds me a bit of some of the mechanistic interpretability work that looked at branch specialization in CNNs and found that architectures which had built in branches tended to have those branches specialize in a way that was consistent across multiple training runs [1]. Maybe the multi-headed and branching nature of transformers adds and inductive bias that is useful for stable training over larger scales. [1] https://distill.pub/2020/circuits/branch-specialization/ observationist 3 months ago | prev | next [–] Neural networks are effectively gauge invariant, and you have a huge space of valid isomorphisms as far as possible "valid" layer orderings go, and if your network is overparameterized, the space of "good enough" approximations gets correspondingly larger. The good enough sets are a sort of fuzzy gauge quotient approximating some "ideal" function per layer or cluster or block (depending on your optimizer and architecture.) https://arxiv.org/html/2506.13018v2 - Here's an interesting paper that can help inform how you might look at networks, especially in the context of lottery tickets, gauge quotients, permutations, and what gradient descent looks like in practice. Kolmogorov Arnold Networks are better about exposing gauge symmetry and operating in that space, but aren't optimized for the hardware we have - mechinterp and other reasons might inspire new hardware, though. If you know what your layer function should look like, if it were ordered such that it resembled a smooth spline, you could initialize and freeze the weights of that layer, and force the rest of the network to learn within the context of your chosen ordering. The number of "valid" configurations for a layer is large, especially if you have more neurons in the layer than you need, and the number of subsequent layer configurations is much larger than you'd think. The lottery ticket hypothesis is just circling that phenomenon without formalizing it - some surprisingly large percentage of possible configurations will approximate the function you want a network to learn. It doesn't necessarily gain you advantages in achieving the last 10% , and there could be counterproductive configurations that collapse before reaching an optimal configuration. There are probably optimizer strategies that can exploit initializations of certain types, for different classes of activation functions, and achieve better performance for architectures - and all of those things are probably open to formalized methods based on existing number theory around gauge invariant systems and gauge quotients, with different layer configurations existing as points in gauge orbits in hyperdimensional spaces. It'd be really cool if you could throw twice as many neurons as you need into a model, randomly initialize a bunch of times until you get a winning ticket, then distill the remainder down to your intended parameter count, and train from there as normal. It's more complex with architectures like transformers, but you're not dealing with a combinatorial explosion with the LTH - more like a little combinatorial flash flood, and if you engineer around it, it can actually be exploited. pizza 3 months ago | parent | next [–] Yes to this. Furthermore: - you can solve neural networks in analytic form with a hodge star approach* [0] - if you use a picture to set your initial weights for your nn, you can see visually how close or far your choice of optimizer is actually moving the weights - eg non-dualized optimizers look like they barely change things whereas dualized Muon changes the weights much more to the point you cannot recognize the originals [1] *unfortunately, this is exponential in memory [0] M. Pilanci — From Complexity to Clarity: Analytical Expressions of Deep Neural Network Weights via Clifford's Geometric Algebra and Convexity https://arxiv.org/abs/2309.16512 [1] https://docs.modula.systems/examples/weight-erasure/ eru 3 months ago | root | parent | next [–] Thanks for the explanations and the great links! srean 3 months ago | parent | prev | next [–] Wouldn't such local invariance tie in with flatness or shallowness of the minima ? This would tie in with the observation that flat/shallow minimas are easier to find with stochastic gradient descent and such weights generalise better. sbinnee 3 months ago | prev | next [–] I was referring to this paper a lot when it was hyped, when people cared about architectural decisions of neural networks. It was also the year I started studying neural networks. I think the idea still holds. Although the interest has been shifted towards test-time scaling and thinking, researcher still care about architectures like nemotron 3, recently published. Can anyone give more updates on this direction of research, more recent papers? choult 3 months ago | prev | next [–] _Fewer_ eru 3 months ago | parent | next [–] https://en.wikipedia.org/wiki/Fewer_versus_less Compare also http://fine.me.uk/Emonds/wholetext.xml tomhow 3 months ago | parent | prev | next [–] Indeed, the original title didn't make that mistake, so we've restored the original title as per the guidelines. mceachen 3 months ago | prev [–] @dang please retitle with (2018)
翻译为中文（简体）
彩票假设：寻找稀疏、可训练的神经网络（2018）（arxiv.org）
3 个月前由 felineflock 发布，获 121 赞同 | 隐藏 | 历史 | 收藏 | 27 条评论
laughingcurve 3 个月 ago | 回复
这是 2018、2019 年的文章了，据我所知，这个假说至今仍只是个假说，而且已有大量反面证据。
swyx 3 个月 ago | 回复上一条
我采访过这篇论文的第一作者乔纳森，他现在基本已经不认这个成果了。https://www.latent.space/p/mosaic-mpt-7b
gwern 3 个月 ago | 回复上一条
你能解释下为什么这么认为吗？我看了里面关于彩票假设的部分，感觉他并没有否定这个假说本身。他通过阿比纳夫表达的、在商业工作中不再继续研究它的原因，只是这类稀疏化方式对硬件不友好（Cerebras 芯片除外）。
“它在普通商用 GPU 上无法为常规商业任务带来加速，所以我在工作中不再跟进、也不想多谈”，这和 “否定彩票假设（认为其错误或不成立）” 差得远了。
oofbey 3 个月 ago | 回复上一条
其实这篇论文刚发表时就很明显了 —— 就算真能找到这类子网络，在真实硬件上也不会更快。我一直对这篇论文评价不高，但它确实让很多人兴奋了一把。
sailingparrot 3 个月 ago | 回复上一条
它之所以令人振奋，是因为它揭示了模型学习的内在机理，无论是否具备商业应用价值。
gwern 3 个月 ago | 回复上一条
Cerebras 就是真实可用的硬件。
oofbey 3 个月 ago | 回复上一条
它确实存在，但从实用角度说几乎没人能用得上。除非你在少数几家拥有这类硬件的机构工作，否则根本不具备现实可行性。
aaronblohowiak 3 个月 ago | 回复上一条
这种情况会持续多久？
oofbey 3 个月 ago | 回复上一条
他们的商业模式很特别。芯片尺寸极大，因此只卖给大客户。而且其制造方式是整片晶圆做成单颗芯片，没有两颗芯片完全一样。通常制造缺陷会导致晶圆部分区域报废、其余按性能分级，而用整片晶圆就只能接受成品的天然差异。所以这注定是个很特殊的平台 —— 每块设备都略有不同。
IshKebab 3 个月 ago | 回复上一条
至少在可预见的未来（比如未来 50 年）都会是这样。
laughingcurve 3 个月 ago | 回复上一条
我亲眼见过它把一位非常厉害的大学教授都给 “学术勾魂” 了。
swyx 3 个月 ago | 回复上一条
他线下基本一直这么说哈哈，不过我可能把这话和我们在神经信息处理系统大会上的后续对话记混了。https://www.latent.space/p/neurips-2023-startups
laughingcurve 3 个月 ago | 回复上一条
太棒了，多谢 —— 我觉得直接听作者本人说会更有说服力。我本来还不太敢发研究相关评论，怕显得太刻薄。
另外，记一下：如果我发了论文又不认，肖恩会来采访我的 :)
yorwba 3 个月 ago | 回复主评论
你想到的是哪些反面证据？我认为这个结论确实没什么实用价值，因为找到 “中奖彩票” 的方法，本质上就是先训完一个大而稠密的模型，相当于买一大堆彩票直到中奖为止，但这并不代表这个观察本身是错的。
kingstnap 3 个月 ago | 回复上一条
这个观察本身也不完全正确。几个月前我看过一个视频，深入探讨了子网络相关问题。
https://youtu.be/WW1ksk-O5c0?list=PLCq6a7gpFdPgldPSBWqd2THZh...（带时间戳）
在该时间点他们讨论到，原论文在国际学习表征大会上的结果只在极小模型上成立，更大的模型并不奏效。要修正这个问题，需要先稠密训练几个轮次，之后再逐步提高稀疏度。
paulsutter 3 个月 ago | 回复上一条
看了这个视频，谢了。
约阿努的意思是，论文中训练稠密网络的方法在非玩具数据集上不成立（论文里早期筛选有效权重的方法并不能提升网络效果）。
但 “彩票假设” 这个词本身，指的是一小部分权重决定模型功能这一真实现象（参考各类剪枝论文）。这个命名很贴切，因为这些权重确实是基于随机初始化的巧合。
被推翻的只是论文基于该观察提出的、训练稠密网络的具体方法。
rob_c 3 个月 ago | 回复楼主
这本质上就是在重述一个结论：训练好的深度神经网络 strongly 依赖初始化参数（这一点很容易证明）。
如果能提前找到这些优质子网络当然很好，但这也恰恰说明纯 DNN 对初始化过于敏感，需要尽量避免。
对比来看，Transformer 对初始化参数的依赖要弱得多。这意味着模型学习能力更强 / 更弱，还是只是训练更稳定？
snaking0776 3 个月 ago | 回复上一条
这个观点很有意思，我之前没怎么想过。有点像 youTuber 做的机械可解释性工作，研究 CNN 中的分支专业化，发现带内置分支的架构，其分支功能在多次训练中表现一致。
或许 Transformer 的多头和分支结构引入了某种归纳偏置，有利于在更大规模上稳定训练。
[1] https://distill.pub/2020/circuits/branch-specialization/
observationist 3 个月 ago | 回复楼主
神经网络本质上具有规范不变性，存在大量有效的同构变换，对应不同的合法层序排列；当网络过参数化时，“足够好” 的近似解空间也会相应变大。这些优质解集近似构成一种模糊的规范商空间，对应每层 / 模块的 “理想函数”（取决于优化器与架构）。
https://arxiv.org/html/2506.13018v2 —— 这篇论文有助于理解网络结构，尤其在彩票假设、规范商、权重置换和梯度下降实际行为方面。
柯尔莫哥洛夫 - 阿诺德网络能更好地体现规范对称性并在该空间上运算，但不适配现有硬件；不过机械可解释性等研究可能会催生新硬件。如果能确定层函数的理想序（如平滑样条），就可以固定该层权重，让网络其余部分在该序下学习。
每层的有效配置数量极大，尤其神经元过剩时，后续层的配置数更是远超想象。彩票假设只是在描述这一现象，却没有形式化：相当大比例的随机配置都能近似拟合目标函数。它未必能帮你榨干最后 10% 的性能，还可能出现无效配置导致训练提前塌陷。
或许存在针对不同激活函数、利用特定初始化方式的优化策略，能提升架构性能 —— 这一切都可以基于规范不变系统与规范商的数论理论进行形式化，不同层配置对应高维空间中规范轨道上的点。
如果能在模型里多塞一倍神经元，随机初始化多次找到中奖彩票，再蒸馏回目标参数量并正常训练，会非常有意思。
Transformer 这类架构会更复杂，但彩票假设并不涉及组合爆炸，更像是小规模的组合爆发，合理设计后甚至可以加以利用。
pizza 3 个月 ago | 回复上一条
完全同意。补充两点：
可以用霍奇星算子方法得到神经网络的解析解 * [0]
用图像初始化网络权重，能直观看到优化器对权重的实际改动幅度 —— 比如非对偶优化器几乎没什么变化，而对偶 Muon 优化器会大幅修改权重，甚至完全看不出原始模样 [1]
* 遗憾的是，内存消耗呈指数级增长
[0] M. Pilanci —— 从复杂到清晰：基于克利福德几何代数与凸优化的深度神经网络权重解析表达 https://arxiv.org/abs/2309.16512
[1] https://docs.modula.systems/examples/weight-erasure/
eru 3 个月 ago | 回复上一条
感谢讲解和优质链接！
srean 3 个月 ago | 回复上一条
这种局部不变性，是不是和极小值的平坦性 / 浅平性有关？
这也能解释为什么随机梯度下降更容易找到平坦极小值，且这类权重泛化能力更好。
sbinnee 3 个月 ago | 回复楼主
这篇论文火的时候我经常引用，那时候大家还很关心神经网络架构设计。也是我开始学习神经网络的那一年。
我认为这个核心观点依然成立。尽管现在研究热点转向测试时代码缩放与思维推理，但研究者仍在关注新架构，比如最近发布的 Nemotron 3。
有没有人能补充这个方向的最新研究进展和近年论文？
choult 3 个月 ago | 回复楼主
应该用 fewer，不是 less。
eru 3 个月 ago | 回复上一条
https://zh.wikipedia.org/wiki/Fewer_versus_less
也可以参考这个对比：http://fine.me.uk/Emonds/wholetext.xml
tomhow 3 个月 ago | 回复上一条
确实，原标题没有这个错误，我们已按规则恢复原标题。
mceachen 3 个月 ago | 回复楼主
@dang 请在标题中加上（2018）
翻译说明
严格保留论坛对话层级（| 回复 对应原回复链），贴合 Hacker News 风格
ML 专业术语统一：winning ticket = 中奖彩票 / 优质子网络，sparsity = 稀疏化，overparameterized = 过参数化，gauge invariant = 规范不变性，inductive bias = 归纳偏置，mechanistic interpretability = 机械可解释性
口语化表达自然本土化：nerdsniped = 学术勾魂，disowns it = 不认 / 否定，toy models = 玩具模型，commercial workloads = 商业任务
人名、机构、会议名保留原文并标注：ICLR = 国际学习表征大会，NeurIPS = 神经信息处理系统大会，Cerebras = 专用 AI 芯片厂商
链接、引用格式原样保留，便于直接对照原文查看