Inexact queries and AI in everything

Google Cloud Summit came to London last week, and we took the opportunity to sit down with database execs Sailesh Krishnamurthy (VP engineering) and Yasmeen Ahmad (product executive Agentic Data Cloud).

The event was wall-to-wall agentic AI, and true to the theme, Ahmad told us that “we’re putting agents at the center … with the goal that humans are not going to be using data platforms in the next three to five years. It’s going to be humans orchestrating agents, and agents actually doing the work.”

One of the key AI-driven changes, Krishnamurthy said, is that when retrieving data “it’s not so much about getting the exact results, but getting the best results.”

For developers skilled in crafting SQL queries that get precise results in the most efficient way, the notion of inexact queries that go through some sort of non-deterministic and compute-expensive parsing may seem like a step backwards.

“If you have exact questions, you need to be able to provide exact answers,” Krishnamurthy told us. “But I think inexact questions are what people are also going to expect. When you think about agentic workloads and operational databases, you want to be able to ask more flexible questions.” An example might be a natural language query that takes into account context, such as previous interactions.

Krishnamurthy described “AI native infrastructure,” including vector indexing, text indexing, and graph technology where “you combine structured and unstructured data, you have to be operating in terms of inexact results and data quality.”

The company is also investing in the “knowledge catalog,” formerly called Dataplex, which is enterprise search now also treated as context for LLMs (large language models). Knowledge catalog aggregates organization data across multiple sources including structured and unstructured sources.

Krishnamurthy said that exact SQL queries are not going away, and that sometimes a “fuzzy question in natural language” might generate an SQL query with exact results.

How do you verify that AI-generated SQL is producing the results you want? “The answer is the same, not just about SQL, but about many AI-related things,” said Krishnamurthy. “The answer is a set of evals you have to maintain …  you might start with something where some results work well and some don’t. And then you have to keep iterating on your blueprints and other pieces of context until your eval set is 100 percent working well.”

By eval set, Krishnamurthy means “a set of questions that are representative tests that users may have, and what is the right query that is generated associated with it, and then a determination of is this query, is this answer correct or not?”

Google SQL as used in its distributed Spanner database, PostgreSQL-compatible AlloyDB, and in the BigQuery data warehouse engine now has AI functions such as AI.IF, which evaluates a condition described in natural language and returns true or false. The prompt value is evaluated using a Gemini LLM; and could return an error or null if the model fails such as when unavailable or out of quota. 

The inefficiency of functions like AI.IF is a problem, but there are possible solutions. One is the idea of proxy models, which Krishnamurthy described as “a tiny model in the database.” A proxy model is trained on the fly, based on a small sample of the data. The query engine evaluates the results from the proxy model, and if good enough, uses it for inference in place of a call to the LLM. According to a paper on the subject proxy models “consume about 400x less tokens, and the latency goes down by 30x-100x.”

We asked Ahmad why she believes humans will soon not interact directly with Google’s data platform. The answer, she said, is based on the idea of intent-driven engineering. “Three years ago everyone was doing prompt training classes. Really, these models were co-pilots or assistants. Now these models are doing multi-step execution, parallel execution, handling complexity. So you can define an intent, a goal, an outcome, and the model will figure out the steps to get there.”

According to Ahmad, humans will act as orchestrators, thinking about business outcomes, and models will do “the hard graft of figuring out the low-level data wrangling.”

She said that today’s staff need to be skilled not so much in prompt engineering, but rather using AI for spec-driven development.  “The focus for the human is getting to the right plan and iterating with the model on what is the right way to think about the problem.”

In business intelligence, she said, companies will move away from dashboards because they only “serve the first layer of predictable questions.” In their place will be “conversational analytics for business users.”

She believes that unwelcome aspects of generative AI, such as hallucinations and prompt injections, are mitigated by improved context, such as from Knowledge Catalog. “I have customers who have got 90 percent plus accuracy with conversational analytics, but that was not the case 18 months ago when the models would get one out of every two questions wrong because they would not have that context.”

A problem here is that even over 90 percent accuracy is not good enough if you are, for example, a customer of a company with heavy AI adoption confronted with a blocked transaction or other rejection because of an inaccurate response. 

Another issue is that injecting AI into every interaction means paying for tokens on top of the base compute and storage resources traditionally consumed by cloud database platforms. Higher productivity and reduced staff costs may more than compensate, but this cannot be taken for granted, particularly as reducing the skill barrier with features like conversational analytics also tends to increase usage.

Giant cloud providers like Google though have plenty to gain. AI, Krishamurthy told us, is driving growth in data storage as well as token usage. He described “a huge overall growth in the business because everyone needs data … Anthropic, for example, rely on BigTable to store all their prompt information. They have other workloads too which are not public.”

Two metrics he is permitted to talk to us about, he said, are that Spanner “now runs 7 ½ billion queries per second at the peak … a year back Spanner might have been 5 billion queries per second.” 

Spanner, he said, “has about 23 exabytes of data. It’s the same with BigTable, roughly 7 billion queries per second and double-digit exabytes.” 

Models make more queries, he said. “Instead of taking the user request and just sending one query, one pattern I’ve seen is a model will send five different queries … it’s hard to say exactly what is happening because the models are trying different things.”®