A single AI agent conversation can look flawless scored on its own and still point to a broken product. That gap is driving a shift in how enterprises evaluate agents, away from scoring individual traces and toward comparing cohorts of users against a baseline.

At VB Transform 2026, Harrison Chase, CEO of LangChain; Hui Zhang, CTO and co-founder of Conviva; and Emmanuel Turlay, director of engineering at CoreWeave, described that shift, along with a parallel move toward cheaper, narrower judge models.

Agent-as-judge — judging one AI agent’s output with another — hasn’t replaced LLM-as-judge, which Chase said remains the default. The larger tension, Zhang said, is between automated judging, whether by LLM or agent, and human review.

“You have scalable but ungrounded, whether it’s agents as judge or LLMs as judge, you grade the outcome, you grade the work. It still is very difficult to ground it and then you use humans and that’s just not scalable,” Zhang said. “The whole industry is facing this, which poison you want to pick.”

Evaluation criteria now function as the product spec

That gap — a conversation that scores well but still signals a broken product — is what teams try to close by building an exhaustive evaluation suite before they ship anything. Chase said that doesn’t work.

“We sometimes see teams that have almost eval paralysis,” Chase said. “They’re like, this is an eval set, I can’t launch it. The best teams launch and then iterate.”

Chase framed evaluation criteria as a living specification, not a one-time test suite: a product requirements document — the standard software-development spec for what an application should do. “Evals are like the new PRD,” he said. “They define what your agent should and shouldn’t do.”

Turlay described hitting the same failure from a different angle. “I was trying to reach 100% coverage for my tests, and I still had bugs in production,” he said — a test suite that looked complete but still missed what mattered, the same gap Chase was describing with evals.

Broad, always-on monitoring, he said, catches more real failures than an exhaustive pre-launch test suite. Teams should set up wide online checks first, use those to identify failure classes as they occur, then build a targeted offline evaluation set around the problems that surface.

Why scoring traces one at a time is a mistake

Even a well-built evaluation process can still score the wrong thing. Zhang’s objection is to how most teams run evaluation: sampling traces, whether 50 of them or a full population, scoring each in isolation. That approach misses a signal that only shows up when comparing cohorts of users against a baseline, a method Zhang calls contrastive analysis.

Zhang illustrated it with a retail example: a shopper asks an agent for a running shoe ahead of a half marathon, the agent asks qualifying questions, and the shopper buys a shoe. Scored individually, that interaction looks fine. But the clarification ratio, how many follow-up questions an agent asks before completing a task, came in three times higher than baseline for that shoe category across the full user population. A second metric, how often shoppers finished their purchase outside the conversation, was five times higher than baseline for the same category.

Neither number is visible from a single trace. Both point to a debuggable, category-specific problem. Zhang said the industry also lacks a second data source: what happens before, between and after the conversation, not just the trace itself.

Sizing the judge to the job

Once contrastive analysis flags which category is actually broken, the next problem is what watches for it going forward — and at what cost. Turlay’s rule was to start with the most capable model available to prove a task is solvable, then work down. If it can’t be done with a top-tier model, he said, it won’t work with a smaller one. Once a pattern proves viable, teams can sample a fraction of traffic instead of judging every interaction, and move simpler tasks like binary classification to smaller open source models.

LangChain took that further, fine-tuning its own model to detect when a user believes the agent made a mistake, a signal Chase calls perceived error. “The model we fine-tuned was a Qwen model,” he said, referring to Alibaba’s open source family. Combining hand labeling with distillation, the result performed well. “Same as [Claude]Sonnet, for, depending on how we served it, either 10 to 100x cost reduction,” Chase said.

Not every guardrail needs a model. Chase pointed to Claude Code’s own guardrails as proof: regexes, the common programming technique for finding and validating patterns in code. “A lot of the guardrails they had were just regexes,” he said. “They weren’t small LLMs, they were just regexes.”

LLM-as-judge doesn’t mean human-in-the-loop disappears

The bigger question is whether using LLM as a judge removes the need for a human in the loop.

Turlay pointed to accountability, drawing on his prior work at a self-driving car company. His team compressed data intake and retraining into a two-week cycle for shipping a new model to the car. Even then, someone still had to sign off.

“I felt confident on behalf of the company to say this model should go into the car,” he said. The same logic extends to legal, finance and healthcare. “Before we can remove a human to say, I endorse this and I take responsibility legally for it, it’s going to be a while before agents can do that on their own.”

Zhang agreed a human has to remain the guardian on corner cases, even as automation eventually runs at a scale that beats individual human accuracy — machines can see more at the pattern level.

Chase went further: that human check isn’t just a safety net. “Human in the loop is really important for building trust in how these agentic systems work, and also really important for memory and learning from systems,” he said. “There has to be interactions in order for the system to learn.”

Robert Downey Jr. returns to the Marvel Cinematic Universe this morning in a role no one saw coming a few years ago. He wears the metal mask of Doctor Doom in the first full trailer for Avengers Doomsday, and the footage makes clear this is no simple villain introduction. Directed by Anthony and Joe Russo, the film lands in theaters on December 18 and pulls together heroes from three separate universes who suddenly share the same battlefield.

Patrick Stewart’s Professor X appears in the teaser with a hushed warning, a sense that something big is about to happen that even a seasoned veteran like him may be unable to stop. By the end of the day, he warns everyone that they’ll have to make a decision they can’t even think about, and that sentence hits home since X has seen enough multiverse mayhem to know when the normal techniques won’t cut it.

Sale

LEGO Marvel Avengers: Endgame Final Battle – Super Hero Building Set for Kids, Ages 10+ – 10 Figures…

• BRING MARVEL STUDIOS’ AVENGERS: ENDGAME TO LIFE – This LEGO Marvel Avengers kids’ building toy is an authentically detailed set for kids to…
• LEGO AVENGERS MINIFIGURES – Captain America, Black Panther, Iron Man, The Scarlet Witch, Ant-Man, Iron Spider, Doctor Strange, Falcon and Chitauri…
• MARVEL ACTION FIGURES – The heroes burst through magic portals into an epic battle with Thanos as kids recreate an iconic movie scene and role-play…

Then Chris Hemsworth’s Thor appears, delivering the trailer’s longest and most urgent speech, attempting to convey his message. He recalls men far stronger than everyone currently standing who died because the threats they faced were more terrifying to him than the one they confront today. He goes on to warn that every single sacrifice those guys made will be in vain until the current group can put aside their old feuds and come together as brothers and sisters. Then he musters the courage to state unequivocally that they will require nothing less than a miracle.

That miracle appears in the form of Chris Evans as Steve Rogers. After years on the sidelines, Captain America returns to the frame, reaches out, and lifts Thor’s hammer as if it were nothing. Thor’s astonished reaction demonstrates how much time has gone since Rogers’ departure and how little some aspects of him have truly changed.

We also get our first good look at Downey’s Dr. Doom in full armor. Then Thor smacks him with a burst of energy, leaving him immobilized. He’s already three steps ahead of any plan you can think of. That sequence underscores Doom’s dominance over all other adversaries in the MCU.

As the trailer progresses, more and more characters from different universes cross paths, with Pedro Pascal, Vanessa Kirby, Joseph Quinn, and Ebon Moss-Bachrach appearing as the Fantastic Four and encountering people that do not belong in their reality. Then there are the original X-Men – Patrick Stewart, Ian McKellen, James Marsden, Rebecca Romijn, Alan Cumming, Kelsey Grammer, and Channing Tatum – who appear alongside Anthony Mackie’s Captain America, Sebastian Stan’s Bucky Barnes, Florence Pugh’s Yelena Belova, Paul Rudd’s Ant-Man, and so on.

Tickets for the exclusive Infinity Vision screenings went on sale the same morning the trailer debuted. This is the final chapter of the Multiverse Saga before Secret Wars, and it appears to be a doozy. The trailer doesn’t explain how these three universes merged or what Doom wants to get out of it; all it does is show the scope of the problem and the unusual group of people who are now standing in his way.

Microsoft is working to fix a known issue affecting Windows Server Update Services (WSUS) servers, which has caused synchronization problems for more than a week.

WSUS was introduced almost twenty years ago to help IT administrators schedule updates for Microsoft products on enterprise networks from a single local update server, rather than updating each endpoint directly from Redmond’s servers.

By default, WSUS syncs with Microsoft Update servers once a day to download the latest metadata for available Windows updates. However, on servers affected by this issue, synchronization is broken, and admins will not be able to deploy the latest Windows updates via WSUS or Configuration Manager.

According to Microsoft, the WSUS sync issues affect both client (Windows 10, version 1607 and later) and server (Windows Server 2012 and later) platforms.

“Organizations might experience increased synchronization times or sync operation timeouts on WSUS servers. This issue began in recent days, with heightened impact observed starting July 13, 2026,” Microsoft said in a Windows health dashboard update.

Microsoft has rolled out mitigation measures on Saturday to prevent WSUS sync times and operation issues from affecting newly installed or rebuilt WSUS servers.

However, it is still working on additional mitigations for previously affected WSUS servers that continue to experience WSUS synchronization issues.

“Synchronization times and sync operations on WSUS servers have been restored and are operating normally for new WSUS installations and rebuilds,” it added. “For WSUS servers that were previously affected, Microsoft is working on mitigation steps to help customers safely remove the affected metadata from their environments.”

One year ago, in May, Microsoft fixed another similar issue after enterprise customers reported being plagued by WSUS errors when trying to update Windows 11 22H2/23H2 systems.

Microsoft addressed another WSUS issue in July 2025 that prevented organizations from syncing with Microsoft Update to deploy the latest Windows updates.

One month later, Microsoft resolved one more issue that blocked the August 2025 security update from being delivered via Windows Server Update Services (WSUS).

Security teams log 54% of successful attacks and alert on just 14%. The rest move through your environment unseen.

The Picus whitepaper shows how breach and attack simulation tests your SIEM and EDR rules so threats stop slipping by detection.

Get the whitepaper

A malicious component dubbed HollowGraph uses the calendar feature in compromised Microsoft 365 mailboxes as a command-and-control channel to receive attacker commands and exfiltrate stolen data.

Researchers analysing the module believe it is part of the Cavern command-and-control framework that has been previously linked to an Iranian threat actor targeting entities in Israel.

At least 12 systems have been infected with HollowGraph, three of them actively communicating with the threat actor between June 3 and July 9.

The collected indicators suggest that the threat actor is focused on organizations in Israel, pointing to a targeted attack for espionage purposes.

Microsoft 365 mailbox abuse

In a report from cybersecurity company Group-IB, researchers say that HollowGraph uses hardcoded details to authenticate to the Microsoft Graph API via a compromised Microsoft 365 account.

The configuration file is stored as logAzure.txt to appear as a regular log file, and “includes the Microsoft Entra ID tenant ID, application (client) ID, client secret, target mailbox address, command-and-control (C2) domain, and two RSA keys.”

The two cryptographic keys are used to encrypt files before delivery to the attacker and to decrypt incoming tasks.

To remain under the radar, the threat actor creates calendar events dated May 13, 2050, with the title in specific formats. Commands and exfiltrated data are concealed within files attached to these calendar entries.

According to Group-IB’s analysis, HollowGraph supports only two commands that let it create calendar entries with stolen files in encrypted form and search for new ones with instructions from the threat actor:

• GET, to search for entries in the format “Event ID: <7-char-taskID>,” download and decrypt received instructions
• SEND, to create a calendar entry in the format “Boss{..}ID{..}” and attach stolen data encrypted with the public RSA key

The researchers describe the mailbox calendar as a “covert dead-drop,” with HollowGraph retrieving commands from events scheduled within a fixed one-hour window between 22:00 and 23:00 UTC on May 13, 2050.

Group-IB explains that the threat actor uses a hybrid encryption scheme that mixes RSA and AES-256-GCM algorithms to secure the communication over Microsoft Graph, keeping inbound and outbound channels cryptographically separated.

HollowGraph has a second, unencrypted communication channel through DNS tunnelling, which is used to receive new Microsoft Entra ID details (Entra ID: tenantId, clientId, clientSecret, and mailbox) to authenticate to Microsoft Graph.

It retrieves the values through IPv6 AAAA record queries to the attacker-controlled domain cloudlanecdn[.]com, and updates the configuration files stored as logAzure.txt.

“Each returned IPv6 address (16 bytes) yields 14 usable payload bytes,” Group-IB explains. The malware assembles the payload from these 14-byte chunks, decodes it as UTF-8 text, and stores the result according to the corresponding configuration field.

While the researchers cannot attribute HollowGraph to a known threat actor, their “analysis identified several technical similarities with the Iranian-nexus threat actor Lyceum.”

However, the available evidence is insufficient to attribute the activity to the threat actor with high confidence. By contrast, the researchers assess with high confidence that HollowGraph is linked to the Cavern framework.

Group-IB suggests that organizations monitor Microsoft Graph and Microsoft 365 audit logs for suspicious application-driven calendar activity, particularly events in the far future, and unusual subjects and attachments.

It is also recommended to look for indicators such as the ‘cloudlanecdn[.]com’ domain and the ‘logAzure.txt’ file, enforce Conditional Access, restrict and audit OAuth client-credential applications, and monitor outbound DNS for tunneling patterns.

Security teams log 54% of successful attacks and alert on just 14%. The rest move through your environment unseen.

The Picus whitepaper shows how breach and attack simulation tests your SIEM and EDR rules so threats stop slipping by detection.

Get the whitepaper

Code references in the iOS 27 developer betas mention an iPhone using multiple batteries. This could be related to the iPhone Fold, but it may have a more pedestrian explanation.

The occasional reference leaks in Apple’s beta operating systems often hint to future features that are on the way. For inbound devices like the iPhone Fold, they can sometimes point to big changes in the design.

According to an examination of the just-released fourth developer beta of iOS 27, MacWorld believes this is a sign that an iPhone with multiple internal batteries is on the way.

The code strings specifically say “The batteries in this iPhone are performing as expected.” Another states “The health of the batteries in this iPhone has significantly degraded.”

There’s also a mention of how “one of the batteries” isn’t a genuine Apple component.

The report highlights the use of plurals in the strings, meaning that multiple batteries are being discussed. Based on these strings alone, it’s entirely possible that they are for two internal batteries in a device.

Stepping back from the hype for a moment, the text may also refer to other situations where multiple batteries are in use. For example, the use of external MagSafe batteries, which basically adds a second battery to a single-battery iPhone.

Folding power

If we take the verbiage as factual, the most probable device that could use two batteries would be the iPhone Fold.

As a folding smartphone, it would use two halves of a body, joined together by the display, wiring, and a hinge. The problem is working out where to place the battery.

If it’s a single cell, the battery would have to be located in one of the two halves of the unit. This would be impractical, partly because most of the weight of the device would be on one side, as well as reducing the potential battery life.

A dual-battery arrangement would maintain battery life, as well as keeping the iPhone Fold balanced. One side would not weigh that much more than the other.

This is also not the first time we have heard of the iPhone Fold using multiple batteries. On July 10, a leaker proposed that there would be two cells in use, “3D Stacked” to get a capacity of around 5,000mAh.

At the time, it was also reasoned that there could be two batteries in use to maximize the available power.

It also wouldn’t be Apple’s first use of dual batteries in an iPhone. The iPhone X was shown in a 2017 teardown to have dual batteries in one body.

Zillow, the real estate technology company, doesn’t get one conversation with its customers. They move from a phone screen to a loan officer to a real estate agent, sometimes over months or years, and expect the context to follow them. A single chatbot could never carry that thread.

At VB Transform 2026, Zillow SVP of Engineering Toby Roberts and Glean co-founder and CEO Arvind Jain described how they built AI architecture meant to carry context across that entire journey — and why context, not raw data, turned out to be the harder problem to solve. Zillow’s products touch roughly 80% of U.S. real estate transactions each year, and the company has been using AI long before ChatGPT existed.

“We pretty quickly identified that we were going to need a persistent context layer that was going to meet our customers and the professionals wherever they were,” Roberts said.

Data was never the hard part

Roberts said Zillow’s AI effort started where most enterprise AI efforts start, with the data itself.

“We started with a large push around making sure our data did have the right foundation,” Roberts said. That meant a data mesh approach, clear data lineage and a governance structure with permissions and identity attached to the data itself.

None of that turned out to be the hard problem. The hard problem was building something that remembered where a customer was in their journey and carried that forward, no matter which surface they showed up on next.

“This context layer has to live to be able to support you where you are at any given point in your journey,” Roberts said. Zillow chose to own that layer itself rather than depend on a single external chat interface, a decision Roberts said the team reached quickly once it looked at the shape of a real transaction rather than a single conversation.

Why Zillow built its own architecture, and where Glean fits into it

Zillow built its own harness rather than route customers through a single model API. The team drew on 20 years of machine learning history behind products like Zestimate, leaning into smaller, task-specific fine-tuned models instead of one general-purpose model.

Internally, that harness runs alongside Glean. Roberts said Zillow now has thousands of Glean agents in production, handling repetitive tasks with tens of thousands of executions across the company. Glean’s pitch, per Jain, is centralizing that integration work once, through the Glean MCP gateway, rather than letting finance, legal and marketing each rebuild their own connections to the same systems.

That centralization is also a cost lever. Jain pointed to two mechanisms: model routing, which sends most tasks to smaller, cheaper models instead of defaulting to frontier models, and precomputed context, which avoids an agent burning tokens assembling its own context from scratch.

“Claude is also very slow because the first part of assembling that context actually takes forever,” Jain said. Routing that request through Glean instead, he said, can cut token consumption by as much as half.

What Zillow and Glean’s approach means for enterprises

Across data, cost and permissions, the session offered a few practical takeaways for enterprises building agentic AI on their own systems.

Build the measurement baseline before the AI push, not after. Roberts said Zillow’s ability to credibly attribute a 40% increase in shipped code to AI adoption rests on a DORA metrics baseline the team put in place years earlier, not on the AI rollout itself.

Centralize context once instead of letting every team rebuild it. Jain’s core argument for Glean’s platform is that duplicated integration work across finance, legal and marketing teams is a hidden cost most enterprises haven’t accounted for.

Don’t assume permission inheritance is enough for regulated data. Even with a permissions-aware context platform in place, Zillow layered hard rules and a standing compliance check on top for its most sensitive categories, rather than trusting the architecture to handle it automatically.

Treat context as a cost lever, not just a capability. Model routing and precomputed context were the two mechanisms Jain pointed to for cutting AI spend, both aimed at reducing wasted token consumption rather than adding new capability.

“Models by themselves are not enough to bring automation with AI inside your enterprise,” Jain said. “You do have to connect it with your enterprise context.”