"We’ve seen a massive shift from convolutional networks to the new Transformer architectures that power today’s [LLMs], but the way these networks route information from one layer to another hasn’t changed all that much." Moulik Gupta explores new research that aims to address this status quo. https://towardsdatascience.com/why-this-decade-old-idea-still-powers-all-of-ai-and-why-its-a-problem/