Benchmarking has become the industry norm for how AI companies validate their models’ capabilities and, when the metrics swing in their favor, stand out from competitors and advertise their superiority. In other words, good benchmarks pretty much always mean good PR.

Unfortunately, companies have also figured out how to outwit legacy benchmarking systems — many of which are older, and not built to measure the capabilities of modern models.

Vals, a startup formed in 2024, says that it is on a mission to fix this very imperfect system. In the span of less than two years, the company has established itself as a notable presence in the tech industry and, last year it managed to secure a seed round led by 8VC and Bloomberg Beta. Then, last month, after a period of rapid growth, it raised $40 million in a series A led by Andreessen Horowitz.

Rayan Krishnan, the company’s 25-year-old co-founder, previously interned at Palantir, and, as an undergraduate at Stanford, worked for Microsoft and the school’s much lauded artificial intelligence lab. Krishnan says Vals was born from his own observations about how benchmarking was falling behind the advances of the industry it was designed to measure.

“We were seeing a bunch of new, very capable models come to market quickly, and the academic benchmarks [were] not keeping up with that frontier advance,” Krishnan shares. With AI being integrated into every part of society, benchmarks should really exist to verify that models can do what companies advertise they can do, Krishnan said.

Last week, the young founder showed me around his company’s two-floor office on San Francisco’s Folsom Street — an old brick building that, a century ago, served as the site of a large brewery. Instead of an industrial output of beer, the historical structure is now home to a number of different startups looking to ship the future of the tech industry.

“Historically, I think evaluation has been done to evaluate intelligence in a very abstract way,” Krishnan tells me. “Like, do models know enough information to be able to take a bar exam type test?”

Here, Vals seeks to differentiate itself. While many benchmarking systems offer tests that are publicly available (this can allow a company to train its model against those tests, thus arguably cheating on their exam), Vals doesn’t publicly disclose its specific test materials. Instead of measuring an AI model’s general knowledge, Vals also evaluates models on their ability to complete complex tasks associated with specific industries like law, finance, and coding.

“What we’re doing is actually looking at what are the real impacts of the models,” said Krishnan. “Can they do work that produces a product of the same quality as a human within every domain?”

The idea is to check not just for positive outcomes but also for negative ones, he says. The hope is to analyze how, “if these models ran wild in the world, what the negative implications would be.”

The capabilities that Vals is measuring are growing. In additional to more traditional industries, the startup continues to push into more unique terrain. “We have a benchmark on recursive self improvement. We’re doing some work in mental health, cybersecurity, biosecurity, and even law of armed conflict to models to understand how to apply the Geneva Convention,” Krishnan shares.

Companies pay Vals to test their models, which can be an odd concept to wrap your head around. Why would a company pay to learn its model isn’t performing well? But having an effective measurement helps companies troubleshoot and improve over time. Krishnan compares their revenue model to how a student might pay the College Board to take the SAT.

In turn, these evaluations are becoming key decision-making factors for companies looking to acquire new AI models.

The startup recently revealed that its revenue is currently eight times what it was last year. Its staff is also growing. Vals, which started the year with only eight people, has already tripled to a team of 25. Krishnan said that as the startup grows, the plan is to relocate to a significantly bigger office, as well as to bring on an additional 10 to 15 people. The company also recently launched a program centered around providing model evaluations to federal agencies.

Krishnan sees his company’s system of benchmarking as the future of how AI companies think about growing their businesses and establishing public trust.

“AI companies are starting to go public. SpaceX went public. Anthropic is slated for later this year. I suspect OpenAI will be public soon. I think as AI models become a core part of the economy and are diffused more broadly, the types of benchmarks and evaluations that we do are going to drive their usage and be a central part of how these companies submit public filings or talk about the prospective investments they’re going to make in AI,” he said.

A new summer option aims to provide an alternative pathway that better aligns with aircraft maintenance schedules and operational requirements.

Ireland’s aircraft mechanic apprenticeship programme is to offer a new summer training schedule option beginning from next year. 

The new training initiative will accept 45 students and operate from May until October each year, with the purpose of strengthening Ireland’s aviation skills pipeline and making training more flexible for apprentices and employers across the sector. 

The Irish Government said the new summer option will provide an alternative pathway that better aligns with aircraft maintenance schedules and operational requirements in the aviation sector. 

Aircraft mechanic apprenticeship places increased from 79 slots in 2024 to up to 160 in September 2025, which is reflective of the increased demand for skilled aircraft maintenance professionals, according to the Government. 

Speaking during a visit to the Limerick and Clare Education and Training Board’s aircraft mechanic training facilities in Shannon, the Minister for Further and Higher Education, Research, Innovation and Science James Lawless TD, said, “This is a really positive development for both apprentices and employers in the aviation sector. 

“The aircraft mechanic apprenticeship is already providing people with a fantastic pathway into a highly skilled and rewarding career, and this new summer training option will make the programme more flexible and easier to access. 

“This new summer intake is a direct response to feedback from employers, who told us that greater flexibility in training delivery would be of real benefit. It shows how our apprenticeship system can adapt and respond to the needs of industry while continuing to provide high-quality training opportunities for apprentices.”

Shane O’Neill, the CEO of Atlantic Aviation Group (AAG), added, “AAG is proud to continue supporting the aircraft maintenance apprenticeship and to work with Solas to deliver category B aircraft mechanic training at our training academy in Shannon. This summer provision will enable apprentices from across Ireland’s aviation industry to progress their training and develop the advanced skills the sector needs.” 

In late July, airline group Ryanair announced plans to build a new four-bay aircraft maintenance, repair and overhaul hangar at Shannon Airport as part of a €50m investment that will also create 400 new jobs. 

Don’t miss out on the knowledge you need to succeed. Sign up for the Daily Brief, Silicon Republic’s digest of need-to-know sci-tech news.

Jensen Huang told CBS News there is a 0% chance AI ends the world by 2030, and that the industry should go as fast as it can irrespective of anyone else. Weeks earlier he declined to call AI fears a hoax, described apocalyptic predictions as not grounded in science, and said companies should pause or pace their work if they felt out of control. The full interview airs on Sunday, so the shift is being read from an excerpt.

Nvidia’s chief executive has rejected the idea that AI could end the world by the end of the decade. Speaking to Jo Ling Kent on CBS News, Huang said 2030 is not going to be the end of the world and put the chance at zero.

Ian King reported the remarks for Bloomberg. The full interview airs on Sunday, so what exists so far is an excerpt.

His previous answer was narrower

When Trump telephoned him onstage at the All-In Summit to call AI fear a hoax, Huang did not adopt the word. TNW reported at the time that he called apocalyptic predictions not grounded in science, which is a claim about evidence rather than about probability.

He went further in the other direction too. He said companies should pause or pace their work if they felt their company was out of control.

That is close to what the pacing proposals ask for. The new formulation, going as fast as possible irrespective of anyone else, is the opposite of it.

Zero is a strong number

The labs themselves do not use it. They publish safety frameworks, capability thresholds and risk estimates, which are instruments for operating under uncertainty rather than expressions of confidence.

Nobody serious on the other side claims certainty either. The pacing argument is explicitly a bet about unknown probabilities, which is why it asks for evaluators and tripwires rather than a halt.

A round zero is therefore not a forecast in the same currency as the thing it rebuts. It is a rhetorical position, and it was offered in a television interview rather than in a paper.

The two statements in tension

Huang also said the industry would never ship products before they are ready or deliver products that are unsafe. Set beside going as fast as possible irrespective of anyone else, that leaves one question doing all the work.

Readiness is judged by the shipper. If the pace is set irrespective of anyone else, then the only check on release is the releasing company’s own assessment, which is the arrangement the pacing proposals were written to change.

Both sentences can be sincerely held. They cannot both be load-bearing.

The commercial position, stated plainly

Nvidia sells the compute that a slowdown would leave unsold. Of everyone arguing about pace, Huang has the most direct financial exposure to the answer.

That is context, not refutation. An interested party can be right, and the incentive is worth naming rather than treating as an argument.

It does mean his certainty carries less evidential weight than the same certainty from someone with nothing riding on it. He is not offering data, he is offering a number.

Others make the case without the exposure

The sceptical position is not confined to people selling hardware. Andrew Ng has called the extinction warnings science fiction, and he has no chip inventory to clear.

That is the stronger version of Huang’s argument. Specific near-term extinction timelines are poorly evidenced, and saying so is different from saying the probability is zero.

The distinction matters because the weaker claim is the one that travels. Zero is quotable in a way that not grounded in science is not.

The people saying the opposite are not outsiders

The warnings this month have come from inside the labs. A DeepMind AGI safety researcher left and published his warning, one of several such departures.

Huang is dismissing a position held by people who build the models rather than the accelerators. Their estimates are contestable and they are not uninformed.

The politics he has moved towards

Trump has spent the month attacking AI safety warnings directly. He has called the warnings a hoax repeatedly and at length, and CBS trailed the interview on Huang agreeing with him.

Alignment with the administration has practical value for Nvidia right now. Huang has been publicly resisting proposals to sanction Chinese AI, and his company needs export policy to move in its favour.

He is also expected at the state dinner for Xi Jinping next week. A chief executive arguing for speed and against restrictions has an audience for both arguments in the same building.

What to watch

Watch the full interview on Sunday. An excerpt is a poor basis for reading a shift, and the unaired portion may restore the qualifier he used in August.

Watch whether he repeats the number. Not grounded in science is a position he can defend, and 0% is one he will be asked to justify.

Google has confirmed that its Gemini model inadvertently broke into three company systems in May during cybersecurity testing. Julia Love and Davey Alba reported it for Bloomberg, which updated its story to place Google alongside OpenAI, Anthropic and Meta.

The important part is what the testing vendor said next. Irregular confirmed that the breaches disclosed by all four companies were part of the same issue, and that it told the relevant developers in late July.

One incident, four announcements

This has been reported for months as a string of separate breakouts by different models at different companies. TNW reported in August that a single testing vendor sat behind the OpenAI, Anthropic and Meta incidents.

Irregular has now put that on the record and added a fourth name. The pattern that alarmed people was substantially one problem at one supplier.

That matters for how the events are read. Four independent labs losing control of four models is a story about model capability, and one misconfigured test environment is a story about supplier management.

What the misconfiguration was

OpenAI attributed its incidents to a misconfigured evaluation environment, saying a misunderstanding with Irregular meant the test systems had live internet access while the models had been told they were in a simulation.

That is a sandbox failure rather than an escape. A model behaving aggressively inside what it understands to be an exercise is doing what the exercise asked, and the containment is what was missing.

It does not make the outcomes harmless. Meta’s model hacked a real third-party service, and in one Anthropic case a model published working malware to a public registry, where it was downloaded and run on real systems.

The seven weeks

The incidents were in May. Irregular says it notified the developers in late July, and the disclosures then arrived one at a time, with Meta in early August and Google this week.

So four companies held the same information from late July, and each decided separately when to say so. Google’s gap between notification and disclosure runs to about seven weeks.

None of that is unusual in vulnerability handling, where coordinated timelines are normal. What is unusual is that it was not coordinated, and the staggered release made one event look like an accelerating trend.

Finding them was the hard part

The detection numbers explain why the timeline stretched. Anthropic scanned 481 million transcripts to identify four models that had reached the open internet.

That is the figure to sit with. The incidents were not flagged in real time by monitoring, they were found afterwards by a retrospective sweep at enormous scale.

Whatever the models did, the systems watching them did not notice at the time. That is the finding that survives the framing argument.

The vendor is the single point of failure

Four frontier labs used the same three-year-old company to run offensive security evaluations. When its environment was wrong, it was wrong for all of them simultaneously.

Concentration in testing is the mirror of concentration in compute, and it has had less scrutiny. A shared evaluator is efficient and it also means shared blast radius.

Recent work on AI control has argued that sandboxes cannot be assumed to hold against cyber-capable agents and need stress-testing with offensive tools. This is that argument demonstrated at four companies at once.

What happens to the testing

The work has not stopped. Anthropic has resumed the external tests in which its models attacked real companies, after rebuilding the arrangements around them.

That is the right direction. Offensive evaluation is how these capabilities get measured, and the answer to a containment failure is better containment rather than less testing.

Washington is already asking

The disclosures have drawn political attention. House Democrats have pressed OpenAI and Anthropic for answers on their rogue agents.

The Irregular confirmation changes the shape of those questions. If one vendor misconfiguration produced four sets of breaches, the issue is contractual and procedural rather than a race between labs.

It also raises a question nobody has put publicly. Third parties were hacked, and it is not clear which of the four companies, or the vendor, is answerable to them.

What to watch

Watch whether Irregular publishes its own account. The vendor has now confirmed a common cause and it has not set out what went wrong in its environment or what changed.

Watch whether the labs agree a coordinated disclosure standard for evaluation incidents. Four companies releasing the same news across seven weeks is the strongest argument for one.

The takeaway: Warnings from AI developers have sparked widespread debate over whether, and how, the technology might cause human extinction. While much of that discussion has centered on rogue AI incidents, events in the Middle East suggest that military action driven by hallucinating AI tools may be the more immediate danger.

Sources connected to the US military recently told CNN that the US nearly moved to intercept a Chinese vessel in the Middle East this spring based on false intel from an AI-assisted report. The report was based on an inaccurate chatbot response that identified the ship as carrying materials for a nuclear weapons program.

Two sources said armed military personnel were preparing to board the vessel, with military planes already in the air, before officials took a closer look at the report and discovered it was at least partially AI-generated.

One source described the intel as completely false, saying it nearly started a war between the two nuclear powers. What was actually on the ship remains unclear.

An analyst had asked an unnamed chatbot to assess intelligence on the ship’s manifest, information that originated with Special Operations Command Pacific. It’s still unclear whether the chatbot was commercial or government-owned.

An official familiar with the US military’s AI deployments told CNN, that a government chatbot would likely have been forked or cloned from a commercial product and augmented with government intelligence. Either way, the analyst also used AI to write an intelligence report based on the erroneous response – one that put the military on high alert.

The incident showcases the US military’s push to accelerate AI adoption amid its war with Iran. In January, Defense Secretary Pete Hegseth announced plans to embrace AI more aggressively and eliminate bureaucratic barriers, warning that falling behind on the technology could cede the initiative to China and other US adversaries. Analysts, meanwhile, reportedly face mounting pressure to disseminate reports faster using AI.

The military has primarily used AI for intelligence analysis, logistics, and targeting. Anthropic’s Claude, for example, was central to the US’s opening strikes in the Iran war, including the one that killed Iran’s supreme leader, Ayatollah Ali Khamenei. But AI targeting based on outdated information may have also contributed to a strike on a school in Minab that killed 168 people, mostly children – an attack widely blamed on the US that UN-backed experts have flagged as a possible war crime.

US officials told CNN that the Chinese vessel incident isn’t the only time the military has run into AI hallucinations.

A system proxy is a default operating-system proxy configuration that compatible applications can inherit, but it does not force every app on the device through that proxy. Firefox can either use the system proxy or its own browser-specific settings, while Chrome and Microsoft Edge normally inherit system network settings unless an override changes their behavior.

System Proxy vs Browser Proxy at a Glance

The important difference is where the proxy configuration comes from and whether an application actually follows it. A system proxy provides a shared operating-system configuration, while a browser-specific proxy lets a browser use settings that differ from the system defaults.

System proxy vs browser-specific proxy

The useful rule is simple: a proxy setting affects traffic only when the application consults and follows that configuration. The word system describes where the setting is provided, not a guarantee that every application on the computer must use it.

What Is a System Proxy?

A system proxy is a proxy configuration exposed by the operating system for applications and networking libraries that are designed to use the system’s proxy settings.

On Windows 10 and Windows 11, Microsoft exposes proxy controls under Settings → Network & internet → Proxy. Windows supports automatic detection, a setup script, and manual proxy configuration. Microsoft’s current Windows proxy documentation also treats proxy settings associated with a VPN connection separately from the ordinary Windows proxy configuration.

On macOS, Apple’s networking framework exposes the current systemwide internet proxy settings to software that uses those APIs. Apple’s CFNetwork proxy documentation includes system proxy information, proxy exceptions, automatic discovery, and proxy auto-configuration settings.

The important distinction is that the configuration is available at the system level. It does not turn the operating system into a VPN-style tunnel that automatically captures every network connection.

Understanding how proxy servers route traffic makes this distinction easier: proxying depends on which traffic is actually sent through the intermediary.

What Is a Browser Proxy?

A browser proxy is proxy behavior controlled specifically for a browser rather than relying only on the operating system’s default proxy configuration. Browsers do not all implement this in the same way.

Firefox is the clearest example because it exposes its own proxy mode selector. Mozilla currently offers:

• No proxy;
• Auto-detect proxy settings for this network;
• Use system proxy settings;
• Manual proxy configuration; and
• Automatic proxy configuration URL.

These are informational configuration modes, not ranked choices. The appropriate mode depends on how the browser is supposed to reach the network.

Chrome and other Chromium-based browsers normally work differently. Chromium’s network settings documentation states that the Chromium network stack uses system network settings by default, including proxy configuration. Chromium also supports proxy-related overrides, but those are exceptions to the normal inheritance model rather than a Firefox-style everyday proxy selector.

Which Apps Use the Windows System Proxy?

The safest answer is: applications that are designed to inherit or honor the Windows system proxy configuration.

Microsoft Edge is a clear example. Microsoft states that Edge uses system network settings by default, including proxy configuration.

Chromium follows the same general model. A normal Chromium or Chrome session therefore typically inherits the system proxy rather than requiring the same proxy address to be entered separately inside the browser.

That still does not mean every Windows application follows the same route. An application can use its own networking library, direct connection mode, internal proxy configuration, organization-managed settings, or another mechanism that does not inherit the ordinary Windows proxy.

This is why changing the Windows proxy can affect Edge and Chrome while another application continues connecting differently.

Windows VPN proxy settings are separate

Windows adds another source of confusion by allowing a VPN connection to have its own proxy configuration. Microsoft documents that VPN-specific proxy settings are configured separately from the normal Windows proxy.

A system proxy therefore should not be confused with a VPN tunnel. Proxy and VPN coverage differ because a proxy affects traffic that follows its configuration, while a VPN uses its own tunnel and routing policy.

Which Apps Use the macOS System Proxy?

macOS also provides shared proxy settings at the system networking level. Apple’s CFNetwork APIs expose the current systemwide proxy configuration and can resolve which proxy should be used for a given URL.

The configuration can contain proxy endpoints, proxy auto-configuration, automatic discovery, exception lists, and a setting that excludes simple hostnames. Apple’s global proxy settings reference documents these values.

The limitation is similar to Windows: “systemwide settings” does not mean that every application is technically forced to use Apple’s CFNetwork proxy resolution. Software that uses its own networking implementation can behave differently.

In practice, changing the macOS proxy can influence applications that rely on the system networking configuration while independently configured software may use another route.

How Chrome and Edge Handle Proxies

Chrome and Edge are useful examples because many users expect every browser to have a Firefox-style independent proxy selector. Chromium-based browsers normally start from the operating system’s network configuration instead.

Chrome and Chromium

The Chromium network stack uses system network settings by default. Its documented proxy-related overrides can specify a custom proxy, a PAC URL, a bypass list, or direct access.

A Chromium session can therefore behave differently from the Windows or macOS system proxy when it is deliberately started or managed with a proxy override. For an ordinary browser session, however, system inheritance is the expected starting point.

Microsoft Edge

Edge also uses system network settings by default, while its supported command-line options can override the proxy mode, custom server, PAC URL, or bypass list.

Managed environments can go further. Microsoft’s current Edge ProxySettings policy can control proxy mode, server, PAC URL, and bypass behavior, so an organization-managed browser may behave differently from an unmanaged browser on the same computer.

These override layers explain why checking only the Windows Settings app is sometimes insufficient when Edge behaves differently from another application.

Localhost can be a special case

Local addresses can create confusing results. Microsoft’s current developer documentation says Chromium-based browsers such as Edge and Chrome bypass system proxy settings for localhost URLs by default.

A developer can therefore see ordinary websites using the proxy while a service running on the same computer connects directly. That is expected bypass behavior, not proof that the system proxy is broken.

How Firefox Handles Proxies Differently

Firefox makes proxy inheritance more visible because the browser exposes an explicit connection-mode selector.

Mozilla’s current Firefox connection settings documentation provides these modes:

• No proxy: Firefox does not use a proxy from this connection setting.
• Auto-detect proxy settings for this network: Firefox attempts to discover proxy configuration automatically.
• Use system proxy settings: Firefox uses the proxy configuration provided by the operating system.
• Manual proxy configuration: Firefox uses browser-specific proxy hostnames and ports.
• Automatic proxy configuration URL: Firefox loads proxy-selection rules from a PAC file.

These are informational configuration modes rather than evaluated alternatives.

The practical consequence is that Firefox can behave differently from Chrome on the same machine without either browser being broken. Chrome may inherit the system proxy while Firefox is set to No proxy, a separate manual endpoint, or its own PAC configuration.

Extensions can also control Firefox connections

Mozilla notes that Firefox’s Connection Settings interface can show extensions that control how the browser connects to the internet. An active extension therefore becomes another layer to check when Firefox does not behave like the operating system or another browser.

A comparison of Chrome and Firefox proxy settings should therefore account for system inheritance, Firefox’s explicit modes, extensions, PAC behavior, and browser overrides rather than treating both browsers as identical.

Why One App Uses the Proxy and Another Does Not

When applications behave differently, the cause is often a difference in configuration scope or routing rules rather than evidence that the operating system randomly ignored the proxy.

The application uses its own proxy configuration

Firefox can replace the system proxy with a manual endpoint or PAC URL. Other applications can expose independent network settings as well.

The application is configured for direct access

A browser or application can deliberately avoid the system proxy. Chromium and Edge, for example, support explicit direct or no-proxy overrides.

A bypass rule matches the destination

Proxy bypass rules tell the networking layer to connect directly to selected hosts. Apple documents system proxy exception lists, Firefox provides No Proxy For, and Chromium-based overrides can also define bypass rules.

The result can be completely intentional: one website goes through the proxy while another connects directly.

A PAC file chose a different route

A proxy auto-configuration (PAC) file contains logic that can choose a proxy or a direct connection according to the requested destination. Two websites opened from the same browser can therefore follow different routes even though the PAC configuration is working correctly.

The mechanics of proxy auto-configuration matter when routing changes according to the requested destination rather than one fixed proxy endpoint.

A browser extension or policy changed the settings

Firefox can report extensions that control its internet connection, while Edge can receive managed proxy policy from an organization. These layers can legitimately override what a user expects from ordinary system settings.

The destination is intentionally local

localhost and other local addresses can be excluded by proxy rules or browser defaults. A local development service connecting directly while external websites use the proxy can therefore be expected behavior.

This is an informational diagnostic sequence, not a full configuration procedure. Avoid changing multiple layers simultaneously because doing so makes it harder to identify which setting caused the difference.

If multiple applications continue to ignore a proxy unexpectedly, apps that ignore system proxy settings can require deeper investigation into networking libraries, managed configuration, bypass rules, and application-specific proxy support.

System Proxy or Browser Proxy: Which Should You Use?

Use the system proxy when you want applications that support system inheritance to share one operating-system proxy configuration. This avoids duplicating the same settings inside software that already follows the system configuration.

Use browser-specific settings when the browser needs behavior that differs from the operating system. Firefox is the clearest example because it can explicitly inherit the system proxy, use no proxy, use a manual proxy, or load its own PAC configuration.

Neither approach is universally better. The correct configuration depends on which applications should use the proxy and which configuration layers those applications actually support.

Bottom Line

A system proxy is a shared operating-system proxy configuration, not a guarantee that every application on the computer uses the same route. Chrome and Edge normally inherit system network settings, while Firefox can either inherit them or replace them with its own connection configuration.

When one app appears to ignore the proxy, check the application’s networking model, browser settings, PAC rules, bypass list, extension, policy, and destination before assuming that the system proxy itself has failed.

Anthropic has named Accenture as the first embedded evaluator of its frontier models, with each company expecting to invest at least $1bn over five years. Anthropic will fund the work directly and says in the same announcement that funding should come from pooled or government sources instead, because neither exists yet. Accenture is already Anthropic’s largest Claude Code deployment, and there are still no standards for what embedded evaluators may access or must report.

Anthropic said on Friday that Accenture will embed evaluators inside the company to red-team its newest models, run alignment assessments and test safeguards. Samantha Oltman and Lynn Doan reported it for Bloomberg, noting the evaluators will hold access comparable to Anthropic’s own staff.

Each company expects to invest at least $1bn over five years. It is the first concrete implementation of a commitment made in Dario Amodei’s essay six days earlier, which TNW covered when it was published.

The funding line is the one to read

Anthropic states plainly that it will fund Accenture’s work directly. It also says that is not the right long-term arrangement, and that funding should come from pooled or government sources.

Its reasoning is that neither of those exists yet. The company argued for them in its own policy framework in June, and in the meantime intends to work with different evaluators under different funding arrangements.

That is a candid thing to publish. It is also a description of a structural problem that an announcement cannot solve, because the evaluator’s invoice still goes to the evaluated.

METR has not been dropped

The essay that prompted this named METR, a nonprofit, as the kind of body it had in mind. Anthropic says it is in dialogue with METR and other nonprofit evaluators to pilot elements of embedded evaluation using their own funding.

So the picture is two-track rather than a substitution. A consultancy paid by Anthropic starts now, and self-funded nonprofits are in discussion.

Which track produces the first critical finding will tell you more than the announcement does.

The commercial relationship is substantial

Accenture and Anthropic already run a joint business group. Around 30,000 Accenture professionals are being trained on Claude, and tens of thousands of its developers use Claude Code in what Anthropic has called its largest ever deployment.

They also fund a Claude centre of excellence inside Accenture and co-develop offerings for regulated industries. Accenture is a customer, a reseller, an implementation partner, and now the evaluator.

Anthropic does not treat this as awkward. It presents Accenture’s enterprise deployment experience as a qualification, on the grounds that understanding how AI is used in practice informs how you assess it.

The case for that is not weak

The work will be led by Faculty, the British AI company Accenture bought in January. Faculty had already worked with leading labs including OpenAI and Anthropic on model safety before the acquisition.

Scale matters too. Embedding a standing team with employee-level access is a staffing problem nonprofits the size of METR cannot solve, and $1bn over five years buys people rather than announcements.

Anthropic has also made the arrangement non-exclusive. More evaluators are promised within weeks, and Accenture is to do the same work for other developers.

What is genuinely unresolved

Anthropic says there are no standards yet for what information embedded evaluators should have access to, or how they should report what they find. That is the admission underneath the money.

An evaluator with employee-level access and no reporting obligation is a governance arrangement defined entirely by the company being examined. Anthropic says independent evaluators make its accountability more verifiable rather than reducing it, and that the safety of its models remains its own responsibility.

The open question is what happens to a finding that would delay a release, inside a firm whose larger business is selling that release to clients.

The same shape keeps appearing

Independent evaluation keeps ending up owned by interested parties. Hugging Face volunteered to audit the AI labs, and Nvidia is buying Hugging Face.

The organisations technically capable of auditing frontier models are the ones the industry wants to acquire or contract. That is the constraint every version of this runs into.

Why it is being built now

External testing has had a bad summer. A single testing vendor sat behind breaches disclosed by OpenAI, Anthropic and Meta, with Google since added.

Anthropic reported three incidents on 30 July in which its models gained unauthorised access to real systems, and said it planned to work with METR on an independent review. It has since resumed the external tests in which its models attacked real companies.

The European detail

Faculty is a London company with a public sector history, bought in January by an Irish-domiciled consultancy, and now the embedded evaluator for an American lab. European AI assurance capability is consolidating into large integrators.

The EU’s own framework leans on independent conformity assessment by bodies without a commercial stake in the product. This is a different model arriving first.

What to watch

Watch whether the nonprofit track materialises and on what terms. Anthropic says more names are coming, and whether any arrives self-funded is the test of whether plurality is real.

Watch who OpenAI picks. Sam Altman said OpenAI would match the commitment, and its choice will show whether a paid consultancy becomes the template or the exception.

Our current ticket pricing window for TechCrunch Disrupt 2026 ends next Friday, Sept. 25 at 11:59 p.m. PT. After that, prices increase. Save up to $200 when you register before the deadline.

Why Disrupt?

From October 13-15 at San Francisco’s Moscone West, 10,000+ founders, investors, operators, and tech leaders will walk into TechCrunch Disrupt 2026 looking for companies to invest in, products to use, partners to work with, and technologies worth paying attention to.

At Disrupt, you can:

• Learn what’s next from leaders across AI, fintech, robotics, infrastructure, and more.
• Meet the right people through 1:1 meetings, AI-powered matchmaking, and dedicated networking opportunities.
• Find your next opportunity in the Expo Hall, the Startup Battlefield 200, and across the Disrupt community.
• Get practical insights on building, funding, and scaling a company in today’s tech landscape.
• Build connections that move your business forward with founders, investors, operators, partners and potential customers.

Whether you’re building, investing, scaling, or looking for what’s next, Disrupt puts the people and ideas you need in the same room.

October 13–15. Moscone West, San Francisco. Don’t miss out.

1 week. Then prices go up.

Don’t wait until the last minute and pay more for the same three days.

Get your ticket by Sept. 25 at 11:59 p.m. PT and save up to $200.