Much of modern AI runs on multiplication. Neural networks behind everything from generated answers to photo organization and song recommendations perform millions or billions of operations that multiply inputs by learned weights. Lizy K. John thinks that’s more work than the job requires.

John, a professor of electrical and computer engineering at the University of Texas at Austin, has spent the past five years working on a class of models called weightless neural networks. Instead of repeatedly multiplying inputs by weights, these networks pass binary inputs through interconnected lookup tables—closer to consulting a collection of stored answers than solving the same arithmetic problem repeatedly. Depending on the task, she says, the networks can be 1,000 times as small or as fast as conventional alternatives while maintaining comparable accuracy.

Her team’s work has so far focused on small, specific problems: medical sensors, activity tracking, keyword spotting. But she thinks the same approach could eventually reach much bigger targets, including the transformer models behind today’s chatbots.

What made you walk away from weights in the first place? Was there a specific moment that pushed you toward lookups instead?

Lizy K. John: A friend casually invited me to a weekly meeting a few years ago to talk about using lookups instead of weights, a technique that wasn’t new. Someone in the UK had built a commercial product around it in the 80s for pattern recognition, and then it just disappeared. A couple of professors at the Federal University of Rio kept working on it quietly for years, just one research group, so it wasn’t a lot of work.

My friend knew I had hardware implementation experience, so he thought I could help make it real. I had a student who I thought was perfectly positioned to take this on, he’d been working on spiking neural networks, another alternative to standard neural networks. The common thread for me was energy efficiency. I was upset by how much power was consumed by AI, and my ears were open for other ways of doing it with less technology, less power.

In less than six months, we had it running on an FPGA, basically a ready-made chip. We were able to create some very small neural networks that used the look up methodology that could fit on tiny chips and didn’t need a GPU to run them. We could get them 1,000 times smaller than what everyone else was getting. That was encouraging.

Energy-Efficient Weightless Neural Networks

Why is now the time to look into weightless neural networks?

John: When you think about current AI models, it’s amazing what they do. I’m impressed by ChatGPT every time I use it, even though it gets things wrong sometimes. But when it’s coming up with the next word in a sentence, it’s doing millions or billions of multiplications to get there. If you ask me a question, I’m not doing multiplications to answer you. I’m thinking, yes, no, I should say this, I shouldn’t say that. And the human brain only consumes about 20 watts of energy doing that.

The model behind most popular networks today is based on a neuron model from a 1943 paper, the McCulloch-Pitts model. The industry took that and expanded it, millions and billions of neurons, to get something that works. But that doesn’t mean it’s needed. That’s my basic thinking. We may have a simpler way of coming up with the same answers.

In the lookup case, we’re essentially looking up a zero or a one, saying go in this direction or don’t. There are no 30-bit or 16-bit weights involved. So no multiplication, which is what takes all the energy, because multiplication is an expensive operation in hardware, and even for human minds. As children, we all struggled to learn the multiplication tables. It’s a hard operation!

What have you been able to demonstrate so far?

John: In data sets like human activity recognition or medical monitoring, ECG, EEG, blood pressure, weightless neural networks can do the job at 1,000 times less energy use. A lot of processing on smart sensors today just collects raw data and sends it to a server or phone, because the processing can’t happen on a battery-powered patch.

Our network is small enough it can sit right on the sensor. The best other small AI model for one problem we looked at is 17 megabytes. Ours is 14 kilobytes, more than 1,000 times smaller. That means no transmitting raw data every millisecond, which is both an energy save and a privacy win, since your data never has to leave the device.

We’ve also shown gains on keyword spotting, the kind of listening a device does before it recognizes “Alexa” or a wake word. The current best industry model takes more than 5,000 nanojoules per inference. We do it in 42 to 79, depending on the variation.

Medical Monitoring and Chatbot AI

If weightless architecture took off tomorrow, what would change first? Chatbots, self-driving cars, phones?

John: Our first target is medical monitoring, something as simple as a bandaid you put on someone for a week, so doctors can know what’s going on without an expensive connected device. Chemistry is another area we’re working in. Students run an experiment, then slowly send data out for offline processing, and there’s error introduced along the way. We want to put the AI right at the point of the experiment, so the answer comes back immediately, like dipping litmus paper and seeing it turn blue.

Can it help chatbots? We’re working on it. A transformer network alternates between an attention layer and a multi-layer perceptron, over and over. We’ve already replaced the multi-layer perceptron part, which is about half of the network. We haven’t replaced attention yet. But eventually, yes, language models are a target.

The arrhythmia detector in this photomicrograph requires only about 10,000 logic gates, compared to the billions on a conventional chip.University of Texas at Austin

There’s also an interesting angle with new kinds of chip manufacturing. We built an arrhythmia detector on a plastic, bendable substrate. That substrate can only fit about 10,000 logic gates, where a modern two- or five-nanometer chip fits billions. A conventional neural network can’t run on something that small. Ours can, because it’s so much smaller to begin with. That substrate is also more sustainable to manufacture, it uses less water than standard chipmaking.

Does the rest of the tech ecosystem need to change to make this work?

John: No. The data can be the same data current models use. If anything we may need less of it, which is an advantage. Training right now still happens on GPUs, just because they’re the most powerful computers available. At some point I’m hoping we can train on FPGA-based hardware instead, because FPGAs already have small lookup tables built into them, so it’s a natural fit for both training and inference. Until then, nothing else in the stack needs to change.

FPGAs have used lookup tables for more than 30 years, but it’s not a big segment of the market, you hear more about other kinds of processors. There have been some recent works in AI that substitute a few multiplications with lookups, but those tend to use one big lookup table, not many small ones interconnected to each other the way ours are. It’s not very common. Those who use it know it’s great, it’s just not popular yet. There wasn’t a dire need before. Now there is.

Far fewer people are working on this than on transformers. Why, and what would it take to change that?

John: I think it’s because when something works, and you can afford to keep doing it that way, there’s not enough reason to move away from it. So far we’ve only been able to show this succeeding on small problems, sensor outputs and similar. For bigger problems, people don’t have the confidence it’ll work, because it seems too simple to scale.

Any time I give a talk on this, at conferences or other universities, anyone who pays attention is really impressed and wants to work on it. But that’s a small handful of people getting attracted here and there. My hope is that we can show it working on a larger language model, and that success might bring more people in. It’s a matter of showing it can be done. I’m hoping.