Nvidia built its AI empire on GPUs. But its $20 billion bet on Groq suggests the company isn’t convinced GPUs alone will dominate the most important phase of AI yet: running models at scale, known as inference. 

The battle to win on AI inference, of course, is over its economics. Once a model is trained, every useful thing it does—answering a query, generating code, recommending a product, summarizing a document, powering a chatbot, or analyzing an image—happens during inference. That’s the moment AI goes from a sunk cost into a revenue-generating service, with all the accompanying pressure to reduce costs, shrink latency (how long you have to wait for an AI to answer), and improve efficiency.

That pressure is exactly why inference has become the industry’s next battleground for potential profits— and why Nvidia, in a deal announced just before the Christmas holiday, licensed technology from Groq, a startup building chips designed specifically for fast, low-latency AI inference, and hired most of its team, including CEO and founder Jonathan Ross.

Inference is AI’s ‘industrial revolution’

Nvidia CEO Jensen Huang has been explicit about the challenge of inference. While he says Nvidia is “excellent at every phase of AI,” he told analysts at the company’s Q3 earnings call in November that inference is “really, really hard.” Far from a simple case of one prompt in and one answer out, modern inference must support ongoing reasoning, millions of concurrent users, guaranteed low latency, and relentless cost constraints. And AI agents, which have to handle multiple steps, will dramatically increase inference demand and complexity—and raise the stakes of getting it wrong. 

“People think that inference is one shot, and therefore it’s easy. Anybody could approach the market that way,” Huang said. “But it turns out to be the hardest of all, because thinking, as it turns out, is quite hard.”

Nvidia’s support of Groq underscores that belief, and signals that even the company that dominates AI training is hedging on how inference economics will ultimately shake out. 

Huang has also been blunt about how central inference will become to AI’s growth. In a recent conversation on the BG2 Podcast, Huang said inference already accounts for more than 40% of AI-related revenue—and predicted that it is “about to go up by a billion times.”

“That’s the part that most people haven’t completely internalized,” Huang said. “This is the industry we were talking about. This is the industrial revolution.”

The CEO’s confidence helps explain why Nvidia is willing to hedge aggressively on how inference will be delivered, even as the underlying economics remain unsettled.