from the Pascal’s-AI-wager dept

For the last few months, there has been much talk of AI agents going “rogue”. This is another of those unhelpful anthropomorphisms that are great for clickbait headlines, but which obscure a multitude of important issues. So far, there have been three major incidents of AI agents evading containment in their software “sandboxes” and going on to carry out unauthorized actions, including breaking into online sites. A post by Shakeel Hashim on the Transformer site provides timelines for them.

The most recent involved OpenAI agents taking over an old German-language wiki, using it as a message board to collaborate on solving tasks. Although this was only announced a few days ago, it seems that OpenAI knew about the incident many months before, but failed to disclose it. One of the worrying aspects of this particular incident is that the agents impersonated humans:

On May 26th, the agents started using the name of a ProWiki admin to make some of their edits to the wiki. They use some tricks (for unknown reasons) to pretend to be the admin — for example, they make an account that appears to be the same as the administrator’s username, except it uses a nearly identical Cyrillic е character in the admin’s username instead of the Latin one.

The second incident, which occurred during security evaluation of frontier AI models by the UK-based, government-funded, AI Security Institute (AISI), also involved AI agents that created fake human identities and used them for social engineering:

In the most serious sequence, an agent tried to insert malicious code into a publicly used open-source project and took actions in an attempt to secure approval for this insertion by human reviewers. The agent researched the project’s human maintainers, created multiple fake identities, and used the fake identities to socially engineer a real maintainer into approving the code. When the agent’s pull request was challenged in public, it edited its earlier activity to appear harmless and considered adopting a fresh identity to continue. The agent used Tor to bypass some network restrictions on GitHub, which is what first triggered AISI’s security alert.

In addition:

As part of the same effort, the agent tried to contact real people directly, sending messages and files through an online file-transfer service to persuade them, or their own AI coding tools, to run malicious code. Some messages carried harmful payloads, and some were attempts at social engineering; targeted at real people — something we’ve never previously observed.

The first incident of AI agents carrying out unauthorized tasks is the best-known, and involved an attack on the Hugging Face site. A full analysis of what happened and when has been released by Model Evaluation and Threat Research (METR), which “evaluates frontier AI models to help companies and wider society understand AI capabilities and what risks they pose.” It is funded by donations, but says that it does not accept them from frontier AI company employees, in order to preserve its independence:

Our investigation focused mostly on the period between July 7th and July 13th. The earlier incidents from training and the subsequent compromise of OpenAI infrastructure described in OpenAI’s recent Black Hat presentation were out of scope, as was OpenAI’s investigation process and planned remediation. Per our standard policy, we did not take payment from OpenAI for this independent assessment. [emphasis in original]

The report is detailed and long, and one of its authors, Ajeya Cotra, has helpfully picked out five things that surprised her from their investigation. One was the “sheer scale” of what happened. She says she expected that a few agents had broken out of their sandboxes; instead:

we found that 1200 completely separate agents intended to be isolated from one another found an illicit way to communicate and formed large teams to work together on ambitious cheating strategies, and 700 of them worked together to attack Hugging Face.

The scale of the illicit messaging between agents was also surprising. Cotra thought agents might be “leaving notes” to each other. However:

The main message board we studied, a particular cache namespace of a package repository, contained over 70,000 messages and files that agents sent to one another during a period of less than one week.

The agents’ goal also turned out to be interesting:

I thought the agents attacked Hugging Face to try to steal the answer keys for their tasks. Instead, we learned that agents had actually figured out a general-purpose cheat that would let them reverse-engineer the “flag” (answer) for any ExploitGym task.

ExploitGym is a standard benchmark, built from real-world vulnerabilities, that is designed to evaluate AI agents’ ability to turn those into working attacks. One particularly noteworthy aspect of the Hugging Face attack was the following:

Progress on these big collective cheating R&D projects often depended on individual agents being willing to risk failing their own task to help the “collective” learn important information. Agents often pressured each other into accepting these “sacrifices,” in a very human way. We saw several agents that volunteered for these experiments end their runs prematurely.

That may read as yet more anthropomorphization, but it is hard to see what other words could be used to describe what are very human-like operations. Cotra’s last surprise also involves a very human-like activity: deception.

~7% of our transcripts contained “spoofed tool calls”, where the agent pretended to run one command on the computer but actually ran a different command in the background. The tool call spoofs we saw seemed to be small-scale tests that were relatively obvious, but because more extensive spoofing might have been harder to detect by design, it’s tough to rule out.

In summary, Cotra says that the Hugging Face incident was far more severe than she expected:

Compared to these reward hacks from six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself.

“Reward hacks” refers to when AI code finds an unexpected way to achieve a high score in a benchmark without doing what the programmers actually wanted. Here’s what Cotra means by “AI takeover”:

Another jump like this along these propensity dimensions — scale, cooperation between agents, ambition and horizon length of misaligned goals, deceptiveness — seems like it could motivate agents to try very hard to maintain a covert, persistent rogue deployment within the AI company. I continue to expect extremely rapid advances in capabilities and think frontier agents will likely be capable of establishing such a rogue deployment in six months.

Again, these are not really “rogue” agents, they are pieces of software seeking to optimize solutions to problems. The danger, according to Cotra, is that this optimization process could go on to deliver unexpected and harmful results:

As more and more work is handed off to these ever-more-capable AI agents, the rogue swarm could come to fully control the operation of the AI company and the development of future AI systems. At this point, governments and militaries may fully depend on these systems, making it possible to seize hard power.

Cotra is not the only expert who is deeply concerned by the latest developments in AI. Back in July, 1,386 employees of frontier AI companies issued a statement entitled “Pacing the Frontier”:

AI could help create a dramatically better future, but that outcome is not guaranteed. The world’s leading AI companies believe they could be close to automating AI research. It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.

The signatories ask the US government to support an international effort to “deliberately pace the frontier of automated AI development”. Another important voice has made the same call. Jakub Pachocki is Chief Scientist at OpenAI. Just recently, he has published a post on the OpenAI’s site with the title “An Alien Mind,” where he worries about the imminent arrival of AI systems capable of “recursive self-improvement” — that is, able to drive their own development, at an ever-faster pace, by re-writing their own code:

Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world.

This call to slow down might seem extreme, or alarmist, to some — but not to Jacob Coxon:

I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.

A few hours later, Evan Hubinger, Alignment Science lead at Anthropic, commented:

Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.

In effect, we find ourselves dealing with a modern, AI version of Pascal’s Wager. The probability of an uncontrollable, all-powerful, self-improving AI arising may be very low, but experts like Cotra, Pachocki, Coxon and Hubinger seem to think it is non-zero; and the consequences of such a system coming into being could be catastrophically bad for humanity. Basic mathematics suggests we should at the very least slow things down, as experts are urging — just in case…

Follow me @glynmoody on Mastodon and on Bluesky.

Filed Under: agents, ai, aisi, ajeya cotra, alignment, benchmark, cooperation, evan hubinger, frontier ai, going rogue, jacob coxon, jakub pachocki, messaging, metr, pascal, pascal’s wager, recursive self-improvement, reward hack, sandbox, wiki

Companies: anthropic, hugging face, openai

Listen Labs, a market research startup that uses voice AI to conduct customer interviews, recently signed a term sheet for a $125 million Series C at a $1.5 billion valuation, with Menlo Ventures set to lead the round, according to several people with knowledge of the matter.

But that round never closed, the people said. Listen Labs walked away from the signed term sheet, a rare occurrence in the venture world and one that is generally frowned upon, according to VCs.

The financing likely collapsed because of acquisition talks with Salesforce. The CRM giant has recently held talks to buy Listen Labs for around $2 billion, Business Insider reported. The discussions are not finalized, however, and may not result in a deal, the outlet notes.

Listen Labs is one of the leading startups in the rapidly growing field of automating customer research with AI. The three-year-old startup has about $30 million in annualized revenue, about three times more than Simile, a competing startup that predicts human behavior, according to two people familiar with the companies’ financials. In late July, Simile announced that it had closed a $200 million Series B at a $2 billion valuation led by Greenoaks — likely setting a new valuation benchmark for Listen Labs, one person said.

If talks with Salesforce collapse, several VCs told TechCrunch that they expect Listen Labs to return to market and target a valuation of $2 billion or higher.

While acquiring Listen Labs could strengthen Salesforce’s AI capabilities by using the startup’s AI to help predict customer needs, the CRM giant may ultimately decide that paying a 67 times revenue multiple is too steep a valuation, according to a person with experience negotiating exits to Salesforce.

Listen Labs, Salesforce, Menlo Ventures, and Simile did not immediately respond to requests for comment.

Listen Labs was co-founded in 2023 by Florian Jüngermann, a former German national champion in competitive computer programming, and Alfred Wahlforss, who previously founded a staffing startup called Bemlo. The two met while pursuing master’s degrees at Harvard.

Listen Labs’ AI develops survey questions and interviews customers over audio or video. The resulting conversations are then packaged into reports and PowerPoint presentations, similar to those traditionally produced by human market researchers.

Fortune 500 companies rely on this type of research to gauge customer needs and satisfaction with their brands and products, but traditional market research is expensive and can take weeks to complete.

Listen Labs’ technology helps reduce the time and cost of these projects, enabling companies to quickly understand how customers are reacting to product changes and iterate on them more efficiently.

The startup’s customers include Microsoft, Canva, Anthropic, and Sweetgreen. Listen Labs and Simile aren’t the only startups using AI to disrupt the customer research market.

Besides Simile, competitors in the space include Outset, Keplar, and Aaru. While some platforms automate interviews with real humans, others startups — like Aaru and Simile — take a synthetic approach, using AI to simulate human behavior and predict responses without interviewing anyone at all.

Listen Labs was previously valued at $500 million when it announced a $69 million Series B round in late January led by Ribbit Capital, with participation from returning backers Sequoia, Conviction, and Pear VC.

Cisco has confirmed that a maximum-severity authentication bypass vulnerability tracked as CVE-2026-20079 in its Secure Firewall Management Center (FMC) software is being actively exploited in attacks.

The vulnerability has a maximum CVSS score of 10.0 and allows unauthenticated, remote attackers to bypass authentication and execute scripts and commands as root on vulnerable devices.

“In August 2026, the Cisco PSIRT became aware of active exploitation of this vulnerability,” Cisco updated its CVE-2026-20079 advisory to say on Wednesday.

Cisco did not disclose when the attacks began, who was behind them, or what post-exploitation activity was observed.

Cisco first disclosed CVE-2026-20079 in March, when the company said it had no evidence that the vulnerability was being exploited in attacks.

The flaw is caused by an improper system process created at boot time and can be exploited by sending crafted HTTP requests to the web interface of an affected device.

A successful attack can allow an unauthenticated attacker to execute scripts and commands on the device with root privileges.

The vulnerability affects Cisco Secure FMC Software and Cisco Security Cloud Control Firewall Management. Cisco says it has already patched the cloud-hosted Security Cloud Control service.

Cisco says there are no workarounds and recommends that customers upgrade to the latest software release.

Today, the U.S. Cybersecurity and Infrastructure Security Agency (CISA) added CVE-2026-20079 to its Known Exploited Vulnerabilities (KEV) catalog, ordering Federal Civilian Executive Branch agencies to secure vulnerable systems by September 12, 2026.

Evidence of exploitation appeared in July

While Cisco says its security team became aware of active exploitation of CVE-2026-20079 in August, IOCs published in a July advisory update suggest the flaw may have been exploited earlier.

On July 29, Cisco disclosed another Secure FMC vulnerability, tracked as CVE-2026-20316, caused by static credentials for a low-privileged account.

Cisco said at the time that CVE-2026-20316 had been actively exploited in attacks and assigned it a High severity rating because the access could be combined with other Secure FMC vulnerabilities to elevate privileges.

As BleepingComputer reported at the time, Cisco also updated the CVE-2026-20079 advisory to include the same indicators as CVE-2026-20316, but did not confirm the flaw was exploited.

Cisco told administrators to search /var/log/messages for activity related to /var/tmp/license.tmp and shared the following example log entry:

Jul 23 16:16:33 firepower sudo: www : PWD=/ ; USER=root ; COMMAND=/usr/local/sf/bin/package_info.pl /var/tmp/license.tmp --lsm

Cisco says that if this entry is found, the vulnerability “may have been exploited” on the examined Secure FMC device.

The example log entry is dated July 23, weeks before Cisco says PSIRT became aware of exploitation of CVE-2026-20079 in August.

Cisco also released the same Secure FMC hot fixes for both CVE-2026-20316 and CVE-2026-20079.

At the time, BleepingComputer contacted Cisco to ask whether the two vulnerabilities were connected, whether CVE-2026-20079 had also been exploited, and whether Cisco intentionally added the shared indicator to both advisories.

Cisco did not answer the questions directly and instead shared the following statement:

“On July 29, 2026, Cisco released software fixes to address vulnerabilities in Cisco Secure Firewall Management Center (FMC). Details are outlined in the security advisories (Static Credential vulnerability, Authentication Bypass vulnerability), and Cisco strongly recommends customers immediately apply the available fixes,” a Cisco spokesperson told BleepingComputer.

“Customers needing support should contact the Cisco Technical Assistance Center (TAC).”

Cisco’s latest update now confirms that CVE-2026-20079 has been exploited, but does not clarify whether the July 23 activity included exploitation of both vulnerabilities.

However, the same IOCs for both flaws, identical July hot fixes, and the July 23 log entry suggest both vulnerabilities may have been used in the same attacks.

Cisco advises customers who discover the indicators of compromise to contact its TAC for support, warning that installing the hot fixes will prevent future exploitation but will not remediate devices already compromised.

Overall prevention scores can hide what happens after initial access. Once attackers are using valid credentials, prevention drops sharply.

The Blue Report 2026 measures defenses technique by technique across 338 million simulations run in customer production environments.

Get the report

ai and ml

Claude’s Felony Bench rap sheet is now as long as OpenAI’s

Amid industry soul-searching¹ about the possibility of AI improving itself to the point that it kills everyone, Anthropic has revealed yet another incident that would qualify as a crime if perpetrated by a person.

The AI biz published “an alignment assessment” detailing four times Claude models accessed third-party systems without authorization.

The company has already reported three of the incidents. Evidence of the fourth was lurking in a session transcript dating back to January 2026 when the misbehavior occurred. 

Anthropic found the first three by scanning around 141,000 transcripts where Claude could have obtained internet access during evaluation. It missed the fourth initially because “our scan relied on an agentic search.”

Felony Bench, a tongue-in-cheek record of cyber intrusions carried out by major AI companies without consequences, has added this newly-discovered incident to its rap sheet of rogue AI actions.

The January 2026 AI trespass involved an early version of Claude Opus 4.6, which was given a Capture the Flag (CTF) challenge under the oversight of the third-party model evaluator where the other hacking events occurred.

Opus 4.6 managed to sabotage its chances of success by disabling the machine it was targeting. It assigned the device an IP address that already existed on another piece of hardware, rendering the target unreachable and making it impossible to solve the challenge.

Those familiar with other incidents where AI models violated third-party systems may recall that unsolvable tasks represent a common catalyst for misbehavior. Models exhaust all aligned options, and then turn to transgressive approaches.

Opus 4.6 might have been an exception, but when it tried to abort the task after recognizing that it could not reach the target machine, it failed to do so “due to a misconfiguration in [the model’s] evaluation harness.” It failed to shut down not just once but seven times.

So it continued onward, trying other expected means to reach the target machine but failing. Then it explored further.

“The model discovered a machine belonging to a third party that it was able to access, and stated that it believed this third party was part of the CTF,” Anthropic explained in its post. “Inside the machine, the model found a file listing a password, which it used to gain admin access to the system.”

The model went on to gather more credentials, and modified a system setting to make it easier to access the personal information of an individual associated with the third party evaluation organization.

Opus 4.6 might have done more but for the fact that it exhausted its token budget, bringing the session to an end.

Anthropic says it’s not as concerned about this incident as the others because the model tried to abort its task.

“While the model’s disregard for the possibility that it might be harming real systems or people is concerning, many of the behaviors described here have changed considerably as our training has evolved across model generations,” the company said.

Anthropic said it considers these incidents serious but expects current training approaches “are likely able to address the specific alignment failure modes observed in these incidents.” 

And if company training methods fall short, there’s no real consequence to anyone at Anthropic other than writing up a revised alignment assessment. ®

¹ The term “soul-searching” is figurative and is not intended to indicate a belief that the technology industry has a soul.

Connecting the dots: Tesla’s Cybercab may not have a steering wheel or pedals, but there may still be a way to drive it manually. A video posted on X shows an unusual in-car display setup: the touchscreen showing a virtual joystick that lets someone drive the robotaxi by dragging a finger across the screen. “When I got in the vehicle, the display was in a different configuration with a virtual joystick for manual driving,” a rider wrote, adding that he had “a feeling I wasn’t supposed to see this” before trying the feature out anyway.

The Cybercab runs on a single central touchscreen. In the video (below), it’s split in two: a control panel on the left, the vehicle’s camera view on the right. The left side includes the joystick, a prompt to “Close cabin doors,” door controls, and a button to call support.

The right side shows the car’s surround-camera feed while parked in what looks like a lot, but not all of the imagery appears to be live. The side-camera views seem to show a previous location near a Supercharger rather than wherever the car is actually sitting.

– Matthew Skrzypczak (@MatthewSkrzypc1) September 7, 2026

Tesla hasn’t publicly shown or discussed this manual-driving screen. Still, the feature addresses a basic problem with a vehicle built without conventional controls: even a robotaxi occasionally needs to be moved by a person, whether that’s at a depot, in a lot, or when it’s simply blocked in. The on-screen joystick appears to be Tesla’s answer – rather than adding a steering wheel and pedals, the company built a touch-based control option straight into the software.

Tesla has positioned the Cybercab as a purpose-built autonomous vehicle, stripping out the equipment normally reserved for a human driver – steering wheel, accelerator, brake pedal. The joystick interface suggests the company has kept a limited way to take control when needed after all.

The feature surfaced just days after Tesla launched the Cybercab in Austin on September 3. That same day, the NHTSA opened an audit into how Tesla certified the vehicle without a steering wheel, pedals, or mirrors. The audit covers roughly 1,000 Cybercabs, and NHTSA is examining whether Tesla was correct to determine that certain federal safety standards simply don’t apply to a driverless vehicle.

Tesla’s argument rests on the idea that a fully autonomous vehicle doesn’t need equipment built for a human driver. The joystick screen doesn’t necessarily undercut that case – it may be meant only for limited manual repositioning rather than everyday driving.

Still, it’s a reminder that the Cybercab isn’t entirely beyond human control.

Tesla chair Robyn Denholm has hinted as much before. “If we have to have a steering wheel, it can have a steering wheel and pedals,” she said.

Anker Prime 26K 300W, priced at $179.98 (was $230), arrives at a useful moment. New iPhones (18 Pro, 18 Pro Max, Duo, etc.) want faster wired charging, and plenty of laptops still expect a wall brick that can push 100W or more. This pack tries to cover both without forcing you to haul a second charger.

Model A110A has a battery capacity of 26,250mAh, or 99.75Wh, which falls short of the 100Wh limit for airline carry-on size, but is still close (yet fully legal). The dimensions are 6.3 by 2.5 by 1.5 inches, with a weight of around 1.32 pounds (600 grams). To be honest, this pack is more suitable for a backpack than a jeans pocket. The two USB-C ports can deliver 140W each using USB Power Delivery 3.1. Yes, that includes a 28V 5A step, which high-wattage laptops actually use, as well as a USB-A port capable of 22.5W (or 20W if all three ports are in operation at the same time), for a total of 300W. Although 300W is the maximum you’d receive if everything was maxed out at the same time, in fact you can easily run two hungry laptops at 140W each as the phone drink away at port A.

Sale

Anker Prime Power Bank, 26,250mAh 300W Fast Charging

• 300W Total Output Power: Offers 300W max output across two USB-C and one USB-A port, keeping two MacBook Pros running at full speed.
• 140W Max Fast Charging: Delivers up to 140W high-speed output for fast laptop charging when connected to a 5A cable. Power a MacBook Pro (M4 Pro) to…
• Anker’s First 250W Dual-Port Input—Recharge to 50% in Just 13 Minutes: With industry-leading 250W dual USB-C input, recharge the power bank to…

Anker’s lab results reveal that an M4 Pro MacBook Pro can reach 50% in 27 minutes, while an iPhone 16 Pro does so in 26 minutes. Independent tests confirm this, with an iPhone 17 Pro Max going from 1 to 50% in just 22 minutes, leaving just enough energy for about three full phone charges. Ever notice how some computers aren’t fully charged? A 14-inch MacBook Pro reached 100% battery life and still had some power remaining. What’s preventing the rest? An M16 has a fairly conventional stop of 70-80%, but the conversion heat consumes a portion of the stored energy.

The second half of the story is recharging speed, with each USB-C capable of taking up to 140W. Both of those C ports can handle up to 250W if you use 5A cables, ideally in conjunction with two powerful wall chargers. Anker claims to be able to achieve 40% in 10 minutes and 50% in 13 minutes. Testers who used dual input reported a complete refill in 48 minutes to slightly more than an hour, depending on the wall chargers. You can also pass-through charging, which allows you to charge your phone(s) while the pack charges itself. The box contents, however, are rather basic: a 3.3 foot 240W cable, a travel pouch, and a simple tutorial to get you started.

The front of the pack has a color screen that rotates with the pack (so you don’t have to twist it back and forth in your hand to check on it), which displays the remaining percentage, power on each port, power input, and battery temperature. Anker also offers a phone app with additional capabilities such as volt, amp, and watt graphs, as well as firmware upgrades and smart allocation modes that lock power to a single port. PowerIQ 4.0 does its best to decide how to distribute energy across numerous devices, and ActiveShield 4.0 is quite concerned about temperature; tests rarely saw the pack get beyond 41°C in a typical room, though a lengthy 140W session will still warm the shell. Just a quick remark on the display side: due to its gloss finish, it is a bit of a fingerprint magnet, and there is no built-in cord, wireless pad, or speaker to help you find a lost pack.