AI Summit Recap: IBM’s Gil on How Enterprises Can Get AI Right

As director of IBM Research and co-chair of the MIT-IBM Watson AI Lab, Dr. Dario Gil oversees one of the world’s largest research institutes. He shapes IBM’s AI vision and, as chair of the National Science Board, has a rare vantage point into the technology’s many evolutions and uses.

As part of The Information’s recent San Francisco AI Summit, Amir Efrati, executive editor of The Information, sat down with Gil to discuss the future of AI, why open AI is better, and what organizations can do to make customizing their models more affordable.

AI’s Small (Model) Future

Not long ago, the narrative around AI was that bigger was better. After OpenAI’s ChatGPT exploded on the scene, tech companies competed for who could pack their large language models with the greatest number of parameters. Gil said that since then the tech world has realized the value of smaller models.

“I’m not of the camp that you can continue to scale compute and data and then nirvana is going to hit. It’s not a simple story of building larger and larger clusters, and that will solve for all of AI’s challenges,” said Gil.

Not only do enterprises not necessarily need the firepower of the largest LLM in the market, Gil noted that inference costs can be prohibitive in the long term. Instead of aiming to train models with 1 trillion to 2 trillion parameters, he said, the new thinking is to train small models with more tokens for specific tasks, which makes AI more affordable to operate.

Why AI Should Be Open

Gil has long been a proponent of open-source AI, arguing that transparency is key in a technology this transformative. He said this is especially true for enterprises looking to train a model on their own proprietary data.

“If you’re going to commingle your crown-jewel data with the data that is used in the AI model, it better be open source, it better be transparent and it better be indemnified to you,” he said.

When Efrati pressed him on the competitive pricing of ChatGPT-4o mini, a closed small model that slashed development costs overnight, Gil noted that training is only part of the AI cost equation, and that enterprises need to think through inference when weighing long-term value.

“The cost structure of inference is a function of the size of the model and how well optimized the stack is to do inference at the lowest possible cost. On both those metrics, open models are doing phenomenally well,” he said.

The Importance of Data Representation

Proprietary data is often the foundation of a company’s competitive advantage. Yet too often, only a tiny portion of that data—1%, by Gil’s estimates—makes its way into the foundational AI model a company uses. If businesses want AI to deliver a true competitive advantage, it is vital that their own data become the backbone of that model.

“I tell my enterprise clients, [if] you take your valuable assets, your data, and in a well-engineered fashion, with the right legal constraints, make sure more of it is represented in foundational models that you will own, it will turn out to be one of your most valuable assets,” said Gil. “The tech world totally gets it, but enterprises are like, ‘How do I do it?’”

Lowering the Cost of Customized Models

So how can enterprises affordably customize their models to increase data representation? Pre-training models from scratch is too time-consuming and cost-prohibitive for most organizations. That leaves two options: retrieval-augmented generation and fine-tuning.

With RAG, organizations create a vector database that stores their private data and lives outside the LLM. Before an LLM returns an answer to a user’s question, it consults this database, grounding the answers in enterprise data. However, RAG isn’t as adaptable as fine-tuning, which allows organizations to adjust the parameters of the model. But because fine-tuning is more specialized, it is also more expensive.

Gil said that thanks to InstructLab, an open-source project launched by IBM and Red Hat, there is a third solution. By allowing people to collaboratively and incrementally add new knowledge and skills to any model, InstructLab lowers the cost of fine-tuning. When enterprises use InstructLab to train smaller models with larger data sets, Gil said they often have better results than even with GPT-4 and Llama.

“You can outperform larger models at a fraction of the cost and build an asset inside your business,” he said.