The Department of Veterans Affairs (VA) Office of Inspector General (OIG) is working toward a hybrid artificial intelligence (AI) platform that would automatically direct tasks to different models based on cost, complexity, and governance requirements, the office’s chief information officer said on Oct. 28.
Speaking at Red Hat’s Government Summit in Washington, D.C., Lance Jenkinson, chief information officer at the VA OIG, said the approach would allow employees to use AI tools without exposing the organization to unpredictable spending as developers deploy increasingly complex AI applications.
“There’s definitely across the federal government a lot of that token anxiety,” Jenkinson said. “I don’t want to have a well-intentioned developer go burn a billion tokens spinning up six agents, and then I get the bill.”
Tokens measure the amount of information an AI model processes and generates. Under token-based pricing, organizations pay according to usage rather than a fixed subscription fee, making expenses harder to predict as adoption grows and employees experiment with increasingly complex AI applications.
To address cost concerns, the VA OIG is working toward a hybrid AI platform that would automatically route tasks to models based on their suitability for particular use cases and the organization’s governance requirements.
Under the planned approach, lower-cost open-weight models could handle routine tasks, such as transcribing interviews. The organization could direct more specialized analytical work to higher-cost models when their capabilities are warranted.
According to Jenkinson, the intent is to avoid requiring employees to understand the cost and performance differences among models or select the appropriate one for every task.
“I really want the platform to be able just to auto-select based on whatever use cases we do in our governance,” Jenkinson said.
The platform represents a desired end state, not a system that Jenkinson said is already operational. He did not specify a deployment timeline or anticipated cost savings.
Ben Cushing, Red Hat’s senior principal chief architect for health and life science, said some federal CIOs have already seen the risks associated with unpredictable AI consumption firsthand.
Cushing said some agencies had seen developers exhaust a year’s planned cloud credits in weeks, leaving IT leaders little room to manage spending against their original budgets.
“They planned out the whole year. They had a budget, and then within a month, a number of developers sure enough burned through the tokens so fast that you know they didn’t have any way to balance their budget,” he said.
Cushing said model routing could help agencies control AI spending by evaluating the cost, subject matter, and complexity of individual requests before selecting a model.
He called the approach “a really strong way to do cost savings,” with queries routed based on “things like cost and the domain, the semantics of the query itself.”