The last big topic in the field of generative machine learning was the leadership drama at OpenAI. Increasingly many people have been saying that no one cares about large language models anymore. They’re right on the money that many moved on to other topics. But does it mean that language models have failed?

Hype train

Hype and consecutively, a hype train (i.e. lots of people buying into some hype; a bandwagon); Wiktionary defines it as “an audience or other group of people who are hyped over something; people who highly and sometimes excessively anticipates something that has been highly advertised and promoted”. People get interested in things or topics because they’re novel, exciting and often have great potential. Hype has a fairly negative connotation — for a good reason.

In the last 10 years, in CS/IT/tech, we had big data, blockchain, and the first wave of machine learning. All of which delivered some value to some adopters. However, they were being sold as catch-all solutions that would solve most if not all of your business problems. Naturally, that wasn’t the case.

It’s large language models’ turn now. Fire your programers, sack your teachers, and lay off your writers. Replace search engines, all documentation and books. Ask the language model to be your therapist, advise you on how you’d progress your career. Or at least that’s what you get if you hang out on Twitter and LinkedIn a lot.

The most bizarre thing is that hype is actually useful. At least to some extent. It allows us to get rid of the fluff quickly — people flock to a particular domain and pick all low-hanging fruits. Then, we can get into the nitty-gritty.

The same happens in research. A new hot topic comes up — a lot of hype, people, lofty proposals and papers. It’s easy to publish on fresh, hyped up topics.

Then, the well dries up and many researchers move on to a new sexy domain. For some research groups, it’s so bad that all they do is chase the hype, instead of building deep expertise in a particular domain (and slowly shifting to new things). In the case of language models, people leave them behind to work on video generation or multi-modal problems.

Unfortunately, how some interpret this dwindling interest is that no one cares about large language models anymore. Pack your bags, and go away, there’s only an above average amount of interest instead of a massive influx.

The reality is that things are getting more difficult now. As they do in all maturing fields of science. Losing the quick pace is discouraging. In particular, if the incentives are structured around printing papers, and graduating students.

Real impact

The boring part of this is that large language models are a great piece of tech. Whether this generation of models or whatever scientific improvement supersedes them.

But… we need to find applications where they are truly useful, instead of just slapping them onto any product and selling it as Ai! PoWeReD bY LaNgUaGe MoDeLs. The result of this is a spectrum:

1. At worst, it’ll be akin to the blockchain fiasco: in theory, great for applications in low-trust, decentralised environments — but in practice, we don’t have many use cases for them in the real world (not right now).
2. Realistically, great semantic search, natural language assistants, writing aids.
3. Optimistically, some disruptive innovation that gives us a new paradigm.

I think it’s easy to differentiate between (1) and (2); however, (2) and (3) might be difficult to tell apart. Is it disruptive enough or still incremental? I guess time will tell.

But to make them into useful products, we need to spend substantial engineering effort, and have real applications. Not research demos touted as production-ready tools.

Crucially, when the field is mature enough, it probably won’t say “powered by large language models” front and center. Take computational photography and object recognition; or malware/fraud/anomaly detection for example. Nowadays, machine learning is a part of all of them. Not a standalone solution.

Next year

In 2024, I want to see some clever products. Ones that mix language models, with some good old engineering and understanding your customer (software development assistants like Copilot are an interesting space). I think it would benefit the perception of large language models. Hype with no success stories? Way to burn out, and discourage future adopters. Nifty tools thanks to clever uses of language models? Yes, please.