INFRA

Inference chip startup Groq raises $650M to grow its cloud platform

Seven months after inking a $20 billion chip licensing deal with Nvidia Corp., Groq Inc. today announced that it has raised $650 million in funding.

Growth investment firm Disruptive and hedge fund Infinitum led the round.

Groq has developed a chip design called the LPU that’s specifically optimized for artificial intelligence inference workloads. In December, Nvidia agreed to license the technologies that underpin the processor. It also hired several key Groq employees, including its founding chief executive.

The transaction produced the Nvidia Grok LPU 3, an inference processor that the chip giant debuted in March. It ships as part of a rack-size, liquid-cooled appliance called the LPQ. The system includes 32 trays that each host three Groq LPU 3 units, one central processing unit and network equipment.

The accelerators in an inference cluster each include a quartz crystal called a clock that regulates processing speeds. Clocks also play an important role in coordinating the flow of data between chips. When accelerators’ clocks move out of sync with each other, data traffic slows down, which negatively impacts AI model response times.

The LPU 3 includes a feature that automatically fixes clock drift to avoid data traffic bottlenecks. According to Nvidia, the chip includes 92 lanes that can each move data to other processors at a speed of 112 gigabits per second. That translates to 2.5 terabits per second of bidirectional bandwidth.

Accelerating the flow of data between chips is not the only way the LPU 3 speeds up inference workloads. The processor ships with 500 megabytes of onboard SRAM, a high-speed memory variety. SRAM is more performant than the off-chip RAM that other AI accelerators use to store data, which translates into faster inference.

Groq operates an LPU-powered cloud platform that companies can use to run inference workloads. The company disclosed today that the platform is processing trillions of tokens per week for 5 million developers.

Groq’s cloud runs across 13 data centers spanning multiple continents. The company will use the proceeds from its funding round to grow its inference capacity with the goal of reaching 200 megawatts by 2027. According to Groq, some of the new processing power will be provided by the LPX, the liquid-cooled LPU 3 appliance that Nvidia debuted in March.

Other cloud operators can theoretically build LPQ-powered inference services of their own. One way Groq could set itself apart from such potential rivals is by extending its platform with new services such as managed databases. Other AI-focused cloud providers, notably CoreWeave Holdings Inc., have also broadened their focus beyond infrastructure to higher-level services.

Image: Groq