It is very hard to guess what an AI compute engine should look like five years into the future, other than the obvious more, more, more. And when AI models are changing faster than chip development cycles, any of the big model builders as well as anyone running AI inference at scale has to have a multi-product, multi-source, and multi-supplier strategy to hedge against any delay or needed feature in a future AI compute engine.

If you squint your eyes and look at Meta Platforms, its business is to run ranking and recommendation engines atop vast social networks that include vast amount of custom content created by users. These R&R applications were collections of algorithms running on vast fleets of CPUs, and eventually they were augmented by machine learning algorithms that ran on GPU accelerators – what are called deep learning recommendation models, or DRLMs.

These DLRMs are very different animals than the large language models created by Google, OpenAI, and Anthropic, and they required very different hardware. DLRMs turn their data into vectors, just like LLMs do, but they create a giant, multidimensional vector space that can be used to show how things might be related – someone on Facebook who likes cat videos and new cat videos added to Instagram, to use an obvious example – without having a graph algorithm that literally links them. In this sense, it is predictive, not deterministic, and that is a very powerful thing for one of the world’s largest advertisers to be able to do.

The trouble is, creating a memory space that is big enough for billions of users doing trillions or tens of trillions of things is a tall order, even when you boil reality down to a bunch of floating point vectors so you can make correlations and presume relationships, it sure do mount up quick. Meta Platforms decided to split the job into two pieces: GPUs with fast HBM hold relevant parts of the embedding tables that are “hot” and the rest are dumped off to CPUs with fat DRAM hold the rest. I went through this in detail with a drilldown on the “Zion” and “ZionEX” and “Grand Teton” hybrid CPU-GPU systems designed by Meta Platforms way back in October 2022, showing how the DLRMs were just as parameter-intense and flops-hungry as the LLMs of the time. I also detailed how Meta Platforms had created what amounts to a memory hypervisor, called “Neo,” that stages parameters and embeddings not only across the CPU and GPU memory hierarchies within a node, but also across a cluster of machines.

What is clear is that Meta Platforms was very good at architecting DLRM systems running R&R training and R&R inference, but it did not control its fate even one little bit when it came to the compute engines it would use. In a very real sense, the advent of the “Grace” CG100 Arm server CPU with fat and fast NVLink ports to share memory coherently with Nvidia “Hopper” H100 GPUs was an effort to keep Meta Platforms in the Nvidia fold. DLRMs needed CPU memory a whole lot more than LLMs do, which do not have giant embedding tables.

The big problem is that DLRMs do not scale up their improvement as more compute is thrown at them, which means that a bigger DLRM is not always a better DLRM. Starting in 2024, however, Meta Platforms started talking about a new approach to DLRMs, which is called generative recommenders and based on a technique called Hierarchical Sequential Transduction Unit (HSTU), that borrowed some of the techniques of LLMs and basically treats user activity as a language and uses generative techniques to predict the next thing you will do, just like an LLM can predict the next token in a sequence based on the corpus of human knowledge. This HTSU method is embedded in the DLRM v3 models Meta Platforms uses across its application platforms.

I happen to think that this insight, to make DLRMs more like LLMs, and whatever algorithms and math underlie the HSTU technique and its generative recommendation engine, are driving the MTIA AI compute engine effort at Meta Platforms. The only reason to create a homegrown AI compute engine is to do the codesign that delivers a radical improvement in bang for the buck. (It is arguably easier to be less costly than a GPU than it is to be better at a given parallel computing task, so this is no mean feat.)

 Meta Platforms recently talked about its MTIA roadmap, putting the kibosh on idle talk that it was floundering and backing up comments made by its chip partner, Broadcom. After pondering the MTIA roadmap a bit, I finally have some thoughts worth writing down. We don’t know much about the future MTIA devices, but you can bet one thing for sure: They will be co-designed to employ this HTSU technique and help drive down the cost of both R&R training and R&R inference. We also think that the HTSU approach means that future MTIA devices could be pretty good at GenAI inference, helping Meta Platforms kill two birds with one architecture stone, supporting its DLRM and its LLM efforts.

Let’s Take A Look Under The Hood

The roadmap reveal by Meta Platforms is here, in case you missed it.

What is immediately obvious is that the future MTIAs look a whole lot more like GPUs and other XPUs than the prior MTIA v1 launched in May 2023 and MTIA v2 launched in April 2023. We have renamed these the MTIA 100 and MTIA 200, and Meta Platforms says that it has deployed hundreds of thousands of these and as well as the MTIA 300, which it has not talked about until now, in its datacenters.

The reason why the future MTIAs as well as the current MTIA 300, which has been deployed for R&R training workloads, need to look like GPUs and AI XPUs because they are shifting away from an application based on embedding lookups and comparisons for user and activity correlations in a vector embedding space to user activity prediction akin to token prediction in an LLM. So of course you need wicked fast memory as well as a lot more compute to use the HTSU method in DLRM v3 versus the machine learning techniques in DLRM v2.

Meta Platforms has been deploying the MTIA 300 compute engines since sometime in the second half of last year. Here are the simplistic block diagrams that Meta Platforms gave out, all strung together side by side so you can see them all at once:

The MTIA 300 is a multichip design, unlike the MTIA 100 and MTIA 200, which were monolithic dies that had compute, I/O, and memory controllers all on the same piece of silicon. The MTIA has a grid of processing elements that connect to banks of what I think is, for economic reasons, HBM3 (not HBM3E) stacked memory. The capacity and bandwidth numbers that Meta Platforms gave out sure make me believe that.

Anyway, as you can see, the MTIA had one compute chip that had HBM3 controllers on it and two I/O chiplets, one top and one bottom, to hook out to the world. Notably, there are a combined dozen lanes of 800 Gb/sec RoCE Ethernet coming off those two I/O chips, which is a fair amount of bandwidth.

As is my habit, I created a table that shows the six different MTIA compute engines side by side so we can compare the past to the future. Take a look and we will talk this out:

As is usual at The Next Platform, anything in regular black text is real data and anything in bold red italics text is an estimate from me.

With the MTIA 300, Meta Platforms shifted from the INT8 processing to FP8 processing, which means data conversion to integer from floating point formats is no longer necessary. The tensor units on the MTIA 300 had a pretty significant performance boost, but they also burned 8.9X as much electricity. Our best guess is that they cost more per 8-bit or 16-bit unit of performance, but that was a given because the MTIA 300 is focused on R&R training and not R&R inference like the MTIA 100 and MTIA 20 were. MTIA 300 was also laying the groundwork for much more powerful MTIA compute engines.

What we do not know is how much vector performance is in the MTIA 300 and its successors, but as our architectural deep dive went into, each processing element has two RISC-V vector cores. Perhaps Meta Platforms will enlighten us?

You can see that the MTIA 400 doubles up the number of compute dies in a socket, and adds a die-to-die link to an SoC that acts as a bridge between the host processor and the MTIA processing elements. We presume that host processor is Arm’s new AGI CPU-1, given that Meta Platforms is the co-designer of that CPU and its first customer. Exactly what this SoC is and does is unclear, but we would have expected it to be wedged between the MTIA compute and its I/O dies, like a DPU of sorts. The MTIA 400 will plug into the “Helios” Open Rack Wide v3 rack that was co-developed by AMD and Meta Platforms and up to 72 of these devices will plug together into a shared memory domain.

Meta Platforms says that the MTIA 400 is done testing in its labs and it is getting ready to deploy it inside of its datacenters. Which has Broadcom, its chip shepherd and possibly its rackscale system builder, very happy indeed.

The MTIA 450 is a fast-following upgrade to the MTIA 400, we think moving from the HBM3E memory used in the MTIA 400 to HBM4 memory. That doubles up the memory bandwidth on the devices to a very respectable 18.4 TB/sec against a 1.75X increase in peak MX4 throughput. It is interesting to note that MX8 and FP16 performance with the MTIA 450 only rises by 16.7 percent compared to the MTIA 400. My guess is that, for whatever reason, this extra performance is latent in the MTIA 400 and MTIA 450 dies and is not being exposed for some reason, perhaps related to yield on 3 nanometer processes from Taiwan Semiconductor Manufacturing Co. Such dark silicon bothers me, particularly when Meta Platforms has no commercial reason to keep sections of the chip dark except if it helps yield.

It is harder to figure what the MTIA 500 is, but it clearly has four chips for processing element compute. I think that moving to four chiplets at the end of next year is a dry run for creating a four-die compute complex ahead of the inevitable shift to High NA processes that come to add more transistors to chips but at a reticle area that is cut in half. If I were doing this, I would use current EUV processes and die interconnects at the 2 nanometer node to perfect getting yield on four-die packages and then be ready for the High NA jump. This would be particularly interesting in that a smaller chip has better yield, which is why AMD shifted to eight-die GPU complexes two years ago. Better to learn this chiplet gluing separate from the High NA jump.

You will note that the MTIA 500 will have versions with 384 GB or 512 GB of stacked memory, which we think based on the specs will be HBM4E. The 384 GB version is very likely a yield play – why on Earth would anyone toss out an MTIA 500 where only 25 percent of its memory is a dud? Broadcom wants to sell it and Meta Platforms wants to buy it. I think there will be a 256 GB variant for the same reason, by the way.

We took a wild stab at estimating the cost of finished MTIA compute engines from Broadcom for Meta Platforms, just to illustrate how quickly the technology and the economics will change. From 2023 to 2027, Meta Platforms will increase the effective throughput of its MTIA devices by 293X (with half of that coming from a move to MX4 4-bit data formats) and as best as I can guess, and its cost per unit of inference throughput, as gauged in peak flops, will have come down by 9.1X.