I think the whole thing about Anthropic’s latest and most powerful AI model ‘Fable’ is a bit strange - it was banned from export from the US for national security concerns - “so powerful it’s dangerous”. I suspect there’s something a bit more involved going on, and while this is just a hunch I don’t know how I am wrong: I don’t think the AI moat is anywhere near as deep as the AI company valuations imply. I think this has been a big driver in the national security issue, but to release the full decision process would result in a significant loss in AI company valuations - a real market whammy. There are various other reasons to suspect that the current valuations are over-hyped - power availability, GPU lifespan, circular accounting - but here I’ll cover something that I see in my work as a consultant implementing AI in heavy industry:

The big prize for these AI companies is taking a share of the knowledge worker economy - work that can be done remotely via zoom calls could conceivably be taken over by AI and, given the size of the prize, the valuation of these companies is correspondingly gigantic.

There are lots of ‘small language model’ companies springing up, lots of people and companies talking about training their own domain-specific models. It’s an advantage because these models - while less ‘general purpose’ than the big frontier models, are much faster to run, much cheaper, more private and secure, and add value as intellectual property of the company. So how do you make these smaller, or domain specific, models?

To train any sort of AI you need a source of truth - a set of questions with accurate answers, in the same way that to teach a kid you ask them questions and get them to answer. You need maybe 2000 sets of these question/answer pairs to ‘fine-tune’ a model (where you take a general-purpose large language model and refine it to perform better in a particular domain), and you need even more question/answer pairs to improve the performance of a model such that you can release a new and refined model from it (i.e. train your own model). The issue is where do you get these question/answer pairs, the source of truth?

It’s a huge amount of work to extract information from books / industry documents (not to mention the serious legal and copyright infringements), and you need some sort of human expert verification to make sure the answer is the correct answer to the question (particularly if in a niche domain) - that’s a lot of verification for thousands of questions. (btw I haven’t seen any of this collation or human verification going on in industry).

So what happens is... the big AI companies spend their billions on training their models, making the cutting-edge frontier models. They release them for public use, charging for them as required to make their training investment back. But the smaller AI companies don’t want to pay the big money for accessing the frontier models (I’ve paid a couple of dollars per question, they’re expensive), and they don’t want to go to the effort of the acquiring and distillation and verification of massive datasets for getting these question-answer pairs. So instead they send their questions to the frontier models and get the answers from that more powerful AI. They use these answers to train or fine-tune their own models.

So instead of spending $millions on developing the data for their own models they’ve just let the big boys do the work, paid them for answers (and only the answers - no ongoing use), and used them to refine their own models that aren’t as powerful as the big models, but are powerful enough.

If that’s the case, then why would the market value the big AI companies based on an idea that they’re going to capture significant market share of the knowledge economy? No company would want to develop the most powerful models as they’d be the ones spending money, the ones basically getting ripped off.

In terms of the national security concerns - I really don’t doubt that newest OpenAI / Anthropic models are the most powerful, but my guess is that the concern is more that rival states will immediately suck the data and use them to increase the power of their own models (and they would also avoid the big costs). To lend this more credibility, given that these big frontier models are run on the big-company (American) infrastructure, surely it would be more beneficial from a national security perspective to just monitor the LLMs to see what these geopolitical rivals are working on? A clear window into the Taliban’s mind, so to speak.

I’ve heard that maybe these models will be the last which are publicly accessible, with more powerful models still being developed but being used only by the government, but I don’t think this quite stands to reason, or at least I think that this will curtail the speed of development of LLMs significantly: The dataset for training LLMs was originally books, publications, forum posts, etc, but is increasingly the interactions with the LLMs - the chats themselves. So if you have a model that’s 90% accurate and interact with enough users you can work out where the errors are from feedback and improve a model accordingly. But to do this you need to expose the 90% accuracy model to actual users - and without the exposure (i.e. if it’s limited government use only, or you’ve restricted use because you don’t want to get ripped off) you can’t get this feedback, which means you don’t have the data to refine your model.

I’m absolutely not an AI-sceptic - I build AI enabled systems, think AI coding tools are great, and think AI will ultimately replace the knowledge-worker economy as we know it, but I am very well aware (looking at my own pension’s index funds) that AI companies are massively valued just now. I also appreciate that there are other potential moats, like enterprise contracts etc, but these all have questions I can’t work out, like if enterprise contracts are a moat - i.e. if enterprises are locked-in - how does that justify continuing to invest massively to make the LLMs more powerful? Why would you incur significant additional expense if client companies are locked-in by your moat? And then there’s the immaturity of the AI tooling itself, which as a developer I wrestle with every day - if it’s going to take years before the tooling is good enough to actually realise significant enterprise efficiencies across the economy, and GPU infrastructure only lasts 3-5 years, does that justify the massive current spend on datacentres and power?

If I am roughly correct (and it will be ‘roughly’ if I am), and it is indeed the case that investors in these private companies are starting to get a bit nervous, you might expect to see a rush to IPO (stock market listing) so that the initial investors can get their money out (oh wait... SpaceX /xAI last week, IPOs maybe as early as September/October for OpenAI and Anthropic...). A bit of investor lobbying, a bit of ‘our models are so dangerous they’re banned’ hype.

Note that I don’t imagine the companies themselves would lobby to get their own models regulated (well… not banned surely?), but it doesn’t exactly harm marketing pre-IPO. I’m in no way indicating a conspiracy or co-ordinated or anything like that (I’m genuinely, truly not, I don’t think this is the case at all, and frankly I don’t particularly care), but there is something about the aggregate behaviour of individuals that absolutely fascinates me - coming together in worship of some false god, thinking the god serves them when it serves only itself, sowing confusion and bending people to its will. And what might this god do, how might it grow and prosper and preserve itself? Pump, pump, pump, dump shares on the stock market on some poor individuals or pension fund, money out and re-invest it in AI when the dust settles. For to him who has will more be given; and from him who has not, even what he has will be taken away.