Nvidia bundles open models to give agents company context

Nvidia says its Nemotron Retriever embedding model raised search relevance by 65% in tests on an open-source OpenSearch setup. That is the main figure behind Agent Knowledge Foundation, which the company added to its Agent Toolkit at GTC Berlin on October 21. The retriever ships with three other components that aim to give enterprise agents the business context they lack, while company data stays where it already lives.

The new set is open source. It bundles Nemotron Retriever, a new Auto-Ontology library, the Kumo family of models for structured data and Nvidia’s CUDA-X data processing libraries. It sits inside the Agent Toolkit that Nvidia first announced at GTC on March 16. The data stays under the access rules companies already run, said Adel El Hallak, Nvidia’s vice president of product for agentic AI. He described the gap to The New Stack: you can connect an agent to Snowflake, but that connection does not show it how the business fits together.

“I wish it was as easy as connected to my Snowflake system, connected to my email exchange, connected to my knowledge base, and then all of a sudden, magically, the AI understands all the data. But the reality is different,” he says. Large language models are “great at predicting next tokens,” in his words. But give one a single spreadsheet with “a whole bunch of tabs” and ask it to find something specific, and “it’s gonna struggle.”

Slower lookups, fewer of them

“In the world of agents, you want the most accurate embedding models or reranking models,” El Hallak says, even if that makes each lookup a little slower. An embedding model turns text into vectors so that similar passages can be found, and a reranker then puts the candidates in order of relevance. With better candidates, the agent gets “much more pertinent information. So the number of lookups at the end, the overall latency actually goes down.”

Nvidia gives three results to support that claim:

• The Nemotron Retriever embedding model raised search relevance by 65% on the open-source OpenSearch setup, needed fewer search steps and added hardly any latency.
• SAP’s fine-tuned version of the reranker ran nearly three times faster.
• ServiceNow got better reranking quality with close to 25% less latency.

Is a tier a plan?

“RAG was awesome, but is it good enough?” El Hallak asks. Retrieval-augmented generation pulls in chunks of text based on how similar they are to the query. A knowledge graph, he argues, does a better job of representing concepts and the relationships between them. Auto-Ontology builds those graphs from a company’s own tables and maps raw data to business concepts such as customers, products and orders. Its listing in Nvidia’s NGC software catalog shows a REST API and a Model Context Protocol (MCP) server that agents can use to query it.

His example is a mobile carrier. “You and I, it’s easy for us to understand. A tier is the same thing as a plan,” he says, but LLMs “mess those up.”

In February, Nvidia bought Illumex, an Israeli startup that automated the creation of semantic layers and ontologies for enterprise data, for a reported $75 million. At the time, an Nvidia spokesperson told Globes that the team’s ontology expertise would help it build “libraries and models that will enable the adoption of Agentic AI in organizations.”

Predictions from tables

Kumo built foundation models for relational data. Nvidia bought the startup in June in a deal The Information valued at more than $400 million. Since then Nvidia has released several Kumo models as open weights on Hugging Face, including Kumo Tabular in September.

El Hallak says the Kumo models replace the per-task pipelines that data teams used to build “for every classification, for every churn, for every sentiment analysis.” In his description the two parts work one after the other: “The whole point of the ontology … is to bring harmonization as you create those knowledge graphs, so there’s [consistency] across the tables, and then you could use Kumo to drive predictive analytics.”

Who holds the permissions

“I’m not looking to compete with, for example, a Snowflake. On the contrary, I want to partner with them,” El Hallak says. The partner list is already concrete. Nvidia’s own supply chain team uses Agent Knowledge Foundation through Palantir’s Foundry platform, building on earlier work between the two companies. Databricks is pairing a Kumo anomaly detection model with Genie Ontology, the context layer it introduced in June. Snowflake customers can run Kumo models in Snowflake Notebooks.

“I’m not thinking about ACLs and data access and all those things. That’s why I need the partners,” El Hallak says. He means access control lists and the other security features that those platforms provide. Nvidia does ship its own controls on the agent side: the toolkit’s OpenShell runtime keeps each agent in a kernel-isolated sandbox. Data access stays with the partners.

SAP buys, Nvidia publishes

SAP went a different way with tabular models. In July it closed its acquisition of Prior Labs, the company behind the TabPFN models, and committed more than €1 billion to it over four years. It now offers TabPFN-3.5 Plus in SAP AI Core alongside its own RPT models.

Nvidia releases its models as open weights. “We just build little weights and models, and we publish them on Hugging Face,” El Hallak says. In September, Nvidia agreed to buy Hugging Face for $12.9 billion.