The market for AI in the SOC has moved faster than the methods for evaluating it.

Just last year, Gartner placed AI SOC Agents at the Innovation Trigger stage with single-digit adoption. 

As of a few weeks ago, Gartner’s “Hype Cycle for Security Operations, 2026” put them at the Peak of Inflated Expectations.

Most AI SOC vendors have a demo that feels like science fiction. Clean alerts go in, and accurate verdicts come out in seconds. It is a compelling pitch.

Accuracy often degrades, though, once these tools leave the curated demo and meet real production conditions. The technology shows promise, and some teams report meaningful gains, but for many organizations the gap between proof of concept and operational reality is still wide.

The guide behind this article puts a number on it: between 80% and 95% of enterprise AI projects fail in production.

To help security leaders close that gap, Prophet Security, a leading agentic AI SOC platform recognized in Rising in Cyber 2026, worked with former Gartner analysts Oliver Rochford and Prateek Bhajanka on a practical, vendor-agnostic guide for evaluating AI in the SOC.

You can download a copy here.

What are you actually evaluating?

A useful question to ask early: Are you acquiring a tool, a capability, or a new way of organizing security work? Be clear on what you expect a proof of concept to prove before you start one.

From Bayesian spam filters to SOAR, automation is nothing new to SecOps. GenAI and large language models are different in scope and reach, applied to everything from detection engineering to evidence gathering and autonomous alert triage, investigation, and response.

That breadth is why alignment between a product’s operating model and your team matters more than it used to, and why it belongs at the center of your evaluation.

1. Can the AI produce reliable verdicts in your environment?

Start with the most important question: can the AI produce accurate verdicts across the scenarios and attack surfaces your SOC actually faces?

The key insight is counterintuitive. Verdict quality does not improve gradually as you feed the model more data. Below a threshold, no amount of fine-tuning or prompt engineering compensates; above it, the model produces reliable verdicts without additional tuning.

The data that pushes quality over that line is usually identity, asset, and organizational context, the information that lets the AI tell an attacker apart from a legitimate administrator.

That has a direct consequence for how you test. A phishing alert can be triaged from email metadata and a reputation lookup. Investigating privilege escalation or lateral movement requires identity data, asset inventories, behavioral baselines, and organizational structure.

If your proof of concept only covers cases where basic detection and telemetry suffice, you are testing the easy scenario and learning nothing about the hard one.

2. Does the operating model fit how your team works?

Misalignment between a product’s operating model and the team using it is one of the most common reasons AI SOC deployments underperform.

A one-person operation leans on AI to do work no one else can, so breadth and cost displacement dominate. A larger team needs AI to amplify human effectiveness, which calls for parallel testing, override telemetry, and deliberate role redesign. The right evaluation is the one built for the team you actually have.

The most revealing test here is human-AI parity: run the system in parallel with your analysts for a couple of weeks, capture baselines before the AI is introduced, and treat analyst overrides as first-class data rather than noise. 

A warning sign is an evaluation where analysts end up ratifying the AI’s conclusions instead of independently reaching their own.

That points to the subtler risk in this category: every AI SOC platform makes a chain of decisions upstream of the analyst: what to ingest, what to suppress, how to prioritize, what context to assemble, and how to frame the investigation. 

The further upstream a decision sits, the less visible it is and the harder it is to reverse. If the AI silently frames every investigation, your human in the loop becomes a rubber stamp. 

This is why explainability and investigation depth matter. Analysts can only trust and audit verdicts when they can see the reasoning behind them.

3. Will the AI stay reliable over time?

A product that works on day one can quietly degrade. This part of the framework tests for durability, and it is the part a two-week proof of concept tends to skip, because it cannot be observed in that window.

The guide flags several areas worth pressure-testing: adversarial robustness, model drift and degradation, adaptability as your environment changes, and lock-in. 

There is always a balance between what a vendor can deliver today, what they envision for the future, and their track record of executing on both. This is where customer references earn their keep, so you can distinguish puffery from reality.

4. What do practitioners wish they had known prior?

The final part of the guide draws on practitioners who have run AI in the SOC in production.

The workforce shift is real, and it arrives faster than expected. 

One enterprise CISO found that roles built around phishing triage and DMARC verification were automated within weeks, before the team had planned what those analysts would do next. The fix is to design the new roles (e.g. detection engineering, threat hunting, red teaming, and AI oversight) before deployment rather than in reaction to it. 

The biggest gains came from expanded scope rather than raw speed. 

They did not come from triaging existing alerts faster; they came from investigating things analysts would never have looked at.

One team resurrected detection rules it had shelved as impractical, correlating HR data, authentication logs, and asset records across locations to catch credential sharing, the kind of work no human would pursue at scale for a low-severity finding. AI makes it feasible. 

It also changes detection engineering economics: experimental detections become viable when AI absorbs the false-positive overhead that used to cost full-time analysts.

“Inconclusive” is a valid answer in its own right. A system that always returns a binary verdict and never says “I don’t know” is masking uncertainty rather than resolving it. Look for tri-state classification (i.e. benign, suspicious, and malicious) with deterministic escalation rules for high-impact decisions.

The bigger picture

A technology at the Peak of Inflated Expectations is nothing to fear. Practitioners just need to manage expectations about what is vendor puffery and what the technology can realistically do in production. 

Validate, ask for references, check case studies, and run your own evaluation. Both the former Gartner analysts and Prophet Security acknowledge that every organization’s needs differ. Some need a service, some need a product, some need a bit of both. There is no one-size-fits-all answer. 

The guide’s throughline is a human-AI hybrid model, one in which probabilistic AI handles triage and investigation while deterministic safeguards and humans in and on the loop govern containment, escalation, and irreversible actions.

Prophet Security is an agentic AI SOC platform that autonomously investigates every alert with transparent, evidence-backed reasoning and escalates the decisions that need a human.

It built its AI SOC analyst around the same principles the guide describes: every verdict shows the queries the AI ran and the evidence it weighed, so analysts can review a complete investigation rather than accept a score on faith, while humans stay in control of high-impact actions.

For the complete four-part framework in this piece, including the scenario-by-scenario context maps, the red-flag checklists, and the full evaluation checklist you can take into a proof of concept, The Hype-Free CISO’s Guide to Testing an AI SOC Solution is available to download.

Get the guide here to access the complete framework, checklists, and questions to ask vendors at each stage.

Sponsored and written by Prophet Security.

Mushroom hunting is cool and fun, but you know what’s not cool and fun? Having slugs or other mushroom hunters discover your secret spot. Or, worst of all, accidentally collecting the wrong kind of mushroom that could make you quite sick—or worse.

Ideally, you’d be able to grow your favorite kind of mushrooms safely in your own home or yard, whether that’s shiitakes, morels, or lion’s manes, just like you would vegetables. But like many gardening-adjacent pursuits, it’s not that simple. Mushrooms, like all fungi, need just three things to grow: spores, a food source, and a humid environment.

However, because mushrooms are incredibly sensitive to light, heat, and contamination, it doesn’t take much to create a prohibitive growing environment. And then there’s the question of whether it’s possible to simulate the critical symbiotic relationships that many mushrooms, especially more coveted ones like porcinis and chanterelles, must maintain with tree roots to thrive.

I am far from a mycology expert, but during the past year, I, a busy full-time-working parent, have engaged in five different attempts to grow mushrooms in my house and backyard: shiitakes on an inoculated log, oysters on a cardboard-enclosed fruiting block, wine caps in spawn buried in bark chips in my garden, oysters from grain spawn in a bucket, and oysters and lion’s manes in a monotub setup.

I laughed, I cried, I made an epic mess of chopped straw in my bathroom, and only two of the five methods succeeded. If you, too, are mushroom-curious and looking to grow your own, here’s what I did—and, in some cases, wish I had done differently.

Also, be sure to check out our guides to mushroom supplements (the functional kind), mushroom gummies, indoor gardening systems, and mushroom gifts.

Which Will Prevail?

Method 1: Mushroom Log

Williams Sonoma Mushroom Log Kit

Read more

Method 2: Spray-and-Grow Box

Back to the Roots Organic Mini Mushroom Growing Kit

Advertisement

Read more

Method 3: Spawn in the Garden

North Spore Organic Wine Cap Mushroom Sawdust Spawn

Read more

Method 4: Fruiting Blocks in a Monotub

North Spore Boomr Bin Automated Mushroom Monotub

Read more

Method 1: Mushroom Log

Williams Sonoma

Mushroom Log Kit

There’s nothing more natural, simple, and affordable than a mushroom log. Many mushroom logs available for purchase are meant exclusively for outdoor use, as they’re just plain old logs with sealed holes full of colonized mushroom spawn. But this 12-inch-tall option from Williams Sonoma is meant for indoor display (though you can also place it outside if you’d prefer), with smooth waxed ends and a rustic metal tag printed with its creation date.

Soak it for 24 hours and—theoretically—you’ll have mushrooms pinning in about 30 days. Even though oyster mushrooms are the easiest variety to grow, for reasons known only to me at the time, I chose a shiitake log, which came with a piece of paper identifying it as a “younger log.” This means the spawn was still in the process of colonizing and should fruit seven to nine months from the date stamped on its tag, which was December 2025. I soaked the log in mid-May and again in early July, but I’m still waiting to see any pins. Williams Sonoma says the log is guaranteed to fruit, so it continues to sit on my dining room table, in a saucer full of water, tented with a plastic produce bag. Some green mold has started to appear on the top, but I remain optimistic, even though this is not my first shiitake mushroom log failure (I tried to grow one from a different brand some years ago, outside under a bush).

U.K.-based healthcare billing software maker Craneware is responding to a cyberattack in which hackers stole a “significant volume” of customer data from its systems, the company said on Monday.

The company said the hackers appear to have been expelled from its systems, but its investigation into the breach is ongoing, according to a statement filed with the London Stock Exchange.

Craneware’s flagship accounting and billing software is used by thousands of clinics, hospitals, and pharmacies across the United States. The company did not say exactly what kinds of data were taken in the breach, only noting that a “percentage” of employee data, customer data, and partner records had been exfiltrated.

The company, whose software helps healthcare providers bill patients for services, handles large amounts of medical records and patient data on behalf of its customers. When it bought Florida-based pharmacy software maker Sentry in 2021, Craneware said it gained access to the company’s 147 million patient records that had been collected over two decades.

Craneware CEO Keith Neilson did not immediately respond to TechCrunch’s questions about the incident, or if the hackers have contacted the company with any demands, such as a ransom. 

It’s not yet clear if the company’s systems can receive email amid the ongoing cyberattack.

While details of the hack are still under investigation, this is the latest data breach in recent months where hackers have targeted tech companies that supply tech and services to the U.S. healthcare sector. By compromising software that many healthcare providers use to analyze and understand their billing processes, hackers can access vast amounts of patient medical and health-related data, and extort the companies with threats of publicly releasing the information.

Craneware is the latest health tech giant to be breached in the past year.

In March, healthcare revenue tech firm TriZetto confirmed hackers stole more than 3.4 million people’s personal and health data from its systems during an earlier cyberattack. That very month, medical data storage giant CareCloud reported a breach of one of its stores of patients’ electronic health records, but has not yet said how much data was taken. 

Last July, medical billing company Episource began notifying at least 5.4 million people that their information had been stolen by hackers.

The largest ever breach of U.S. medical and healthcare data occurred in 2024, when a Russian-speaking ransomware gang hacked UnitedHealth-owned Change Healthcare. The hackers stole the medical and patient records of at least 192 million people, which the company conceded affected a “substantial proportion of people in America.”

The debate over where the line should be drawn between privacy and public safety has only intensified in the AI era. And Flock Safety, a company that has seen both an influx of investment and an intense public backlash, sits right at the center of that debate. 

That’s why at TechCrunch Disrupt 2026, we’re bringing Flock’s founder and CEO Garrett Langley to the stage to speak about those very issues. Disrupt has always had its fair share of hard-hitting conversations, and you should expect no less from this session. 

And Langley’s discussion of AI, surveillance, and the future of public safety is just one of the many compelling conversations happening at Disrupt this year, which will take place October 13-15 at San Francisco’s Moscone Center. You can dive into the programming we’ve announced so far here, or take advantage of lower, early ticket prices right here. 

How Flock Safety, and Langley, got to Disrupt 2026 

Atlanta-based Flock Safety isn’t Langley’s first experience within the startup space. In past roles, he worked on the car subscription service Clutch and the platform Experience, where he oversaw several teams, including engineering and design prior to its sale to Cox Enterprises for $200 million in 2014.  

Flock started in 2017, with Langley partnering with co-founders Matt Feury and Paige Todd, going through Y Combinator before experiencing significant success selling its surveillance technology to businesses and law enforcement agencies. You may associate the company with license place recognition, though it has more general surveillance use cameras and has been developing drone first responders following its acquisition of Aerodome. 

The company’s most recent fundraise last year, a $275 million round, placed Flock at a $7.5 billion valuation, which was a sharp increase from the prior year’s $4.8 billion valuation. Altogether, the outfit has secured $950 million in total funding since inception. 

Flock’s business success has been paired with pushback from privacy advocates like the ACLU and members of the general public, with some people going so far as destroying Flock’s surveillance cameras as it continues to offer its cameras and databases to U.S. Immigration and Customs Enforcement in particular. Among the biggest apparent setbacks for Flock, Amazon’s Ring recently ended a partnership with the company, and the LAPD let its deal with Flock lapse this month.  

Langley will discuss the company’s approach to the increasingly interwoven nature of AI and the surveillance industry in a discussion that’s certain to prove illuminating to Flock’s supporters and its critics. 

To watch the discussion in person and join the other 10,000 founders, investors, tech enthusiasts, and innovators who will be flocking to San Francisco for TechCrunch Disrupt 2026 on October 13-15 for a wide array of workshops, speaker talks, after-parties, and networking, snag a ticket at the best price you’ll be seeing for the rest of the year right here. 

WTF?! Amazon has a reputation for squeezing customers for all they’ve got, but increasing billing prices from a couple of cents to $1.5 trillion really shows how Jeff Bezos got so rich. Thankfully, though, this was just an error by Amazon Web Services, but it still caused one customer’s soul to leave their body, in their own words.

Bill Radjewski, who runs CollegeFootballData.com, received the sort of email most of us would rather not receive last week: an AWS alert informing him that he had accrued $1.5 billion in usage fees. Should he manage to win several lotteries and somehow pay it, he was still expected to face a $3 billion bill on August 1.

During the six years he’d been with AWS, his account had never exceeded $0.02 – most months his bill was just $0.01.

However, those figures were rookie numbers compared to what Dan Harvey, the head of marketing at the Hampshire-based Learning Through Landscapes, was billed.

Harvey told the Guardian that after his previous month’s AWS bill came to 43 cents, he was especially surprised to find a bill for $7.8 billion. “I had to have a real dig around with our tech support team, while I was in full panic mode, trying to find what was going on in our account,” he said.

From Learning Through Landscapes

There was also a student in Delhi whose regular monthly AWS bill of $1.28 increased to $10.9 billion.

Another user, Bharath, saw their bill rise an impressively large 745,728,201,771% from the previous month to $1.5 trillion. “I just saw $1.5tn on my AWS bill and my soul left my body,” they wrote.

– Bharath (@Bharath_uwu) July 17, 2026

But the record seems to belong to the user who racked up a $7.1 trillion bill in service fees since July 1, which is more than double Amazon’s market cap.

Amazon said that after 30 minutes of investigating, it determined that the root cause was “an issue with unit pricing within the estimated billing computation subsystem.” The company provided no further details but paused the bill estimation system.

AWS later said it was “rolling back a recent change to the billing computation subsystem” and attempting to restore the last known good version of its estimated billing calculations.

It appears that the issue wasn’t an easy fix. A few hours after that initial message, AWS wrote that “Our efforts to backfill corrected estimated cost and usage data are still underway. We are progressing slower than anticipated.”

While most people obviously knew this was an error on Amazon’s part, some have pointed out that this could really have alarmed customers and may have even caused health issues. “Good morning to everyone enjoying their heart attacks,” wrote X user Gerred

Even Amazon tried to make light of the situation, though it’s easy to imagine that not everyone was laughing about it . The company wrote on X, “Typo alert: Some customers saw quadrillion-dollar AWS billing estimates today. Slight miscalculation on our end (very slight ). We’re fixing it now. No action needed on your end. Sorry for the confusion. Real question: what will you do with those trillions instead?”

SYSTEMS

Redmond has stabilized new installations, but existing deployments remain stuck in sync purgatory

Microsoft’s aging Windows Server Update Services (WSUS) has hit “severe degradation,” leaving some organizations facing painfully slow synchronizations or timeouts when fetching updates.

According to Microsoft, “a buildup of publishing metadata” has meant that “organizations might experience increased synchronization times or sync operation timeouts on WSUS servers.”

We’re told the problem isn’t related to the recent Patch Tuesday update, but Microsoft noted “heightened impact observed starting July 13, 2026.”

Administrators use WSUS to deploy and manage the distribution of updates and features in an enterprise environment. It was deprecated some time ago, but remains a supported part of Windows and, as such, receives fixes and security updates, even though there will be no new features.

Microsoft’s advice in October 2024 was to move to an alternative, such as one of its cloud-based options. However, administrators tend to opt for an “if it ain’t broke” approach and the service was still flagged as supported and suitable for production deployments.

However, failed synchronization could delay organizations’ ability to test and deploy patches, leaving systems exposed for longer.

Microsoft has not yet published a workaround for affected servers. For new and rebuilt installations, it deployed a mitigation on July 18 that restored normal synchronization, noting: “This mitigation prevents newly installed or rebuilt WSUS servers from encountering this issue.”

For existing affected WSUS servers, though, “Microsoft is working on mitigation steps to help customers safely remove the affected metadata from their environments. We will provide more information when this guidance is available.”

Microsoft’s affected platform list spans pretty much every desktop and server operating system still supported by WSUS, from the newest and shiniest Windows 11 26H1 all the way back to Windows 10 1607 and Windows Server 2012.

There isn’t much administrators can do at this stage other than wait and see, since the problem appears to be on Microsoft’s end. At present, it could be categorized as a headache, and perhaps a symptom of Microsoft’s inattention to the service.

However, should it linger and begin having a more severe impact on an enterprise’s patch cycle, action will be required. ®

from the laws-are-just-suggestions dept

Brendan Carr and the Trump FCC are finalizing plans to illegally eliminate what’s left of the country’s already barely functional media consolidation limits; a specific gift to Trump-friendly right wing broadcasters that are hoping to monopolize what’s left of local U.S. broadcast news so they can more efficiently spread propaganda and kiss the president’s ass.

Current laws (remember those?) prohibit any single local broadcast news company from serving more than 39 percent of all TV households in the US. The original (good) idea was that this helped protect opinion diversity and competition in the local broadcast news space. Republicans don’t like that, because they want to replace all journalism with right-wing and oligarch friendly propaganda.

Brendan Carr last March had already made it clear he viewed the law as optional when he granted Nexstar Media Group a waiver for its $6.2 billion acquisition of Tegna. That deal would let the company reach more than half of all U.S. households with what passes as “local news.” Now he’s trying to replace a congressionally-approved law with a “case by case review” dictated by Republican whims:

“Carr now plans to repeal the 39 percent limit and replace it with a “case-by-case review” of each proposed merger, the chairman announced today in an op-ed published on Breitbart. The change would make it easier for the FCC to pick and choose which station groups get to surpass the limit. Under Carr, this would likely benefit news companies that provide favorable coverage for President Trump.”

This is, to be clear, illegal. Something the FCC’s lone Democrat, Anna Gomez, made clear in her own statement:

“This unlawful effort to hand control of the public airwaves to billionaire buddies of this
administration will destroy local newsrooms, silence community reporting, and drive-up
costs for the American families who depend on local stations for news and emergency
alerts. A free and diverse media landscape depends on real limits on how much of the public
airwaves any one company can control, and this FCC is now poised to allow local
broadcasters to sell those airwaves off to the highest bidder. Congress set the 39 percent
national ownership cap in federal law, and only Congress has the authority to raise or
eliminate it. The Commission cannot waive away that limit simply because these corporate
behemoths want to get out from under it.”

Clearly there will be lawsuits, though they’re likely to drag on until long after Nexstar and Tegna have merged, with future regulators being very unlikely to unwind the transaction. I’d then expect to see Sinclair Broadcasting to merge with the remaining company, creating a monopoly over local broadcast TV.

While people are quick to insist that “who cares, nobody watches this stuff,” they don’t seem to realize that somewhere around 80 million households still watch local broadcast TV channels via antenna, cable TV, streaming providers, or satellite.

As I’ve frequently discussed, most of these local broadcasters deliver a sloppy combination of lazy infotainment and right-wing agitprop, as that viral video about Sinclair Broadcasting made clear a few years back:

Carr’s very unsubtle goal here is to turn the entirety of U.S. local broadcast television into propaganda arms of the U.S. right wing. That’s not an opinion or hyperbole, and he’s well on his way already. It might help if there was a functional opposition party that had made meaningful media reforms a centerpiece of their political platform anytime in the last quarter century.

And this is, of course, just local broadcast TV. We’ve also got Carr’s FCC helping Larry Ellison do the same thing to CBS and CNN. Ellison’s also steadily doing the same thing to TikTok while Elon Musk does the same thing to what used to be Twitter. If you stand back, tilt you head, and squint just right, you might begin to notice a consistent theme.

Filed Under: brendan carr, broadcast tv, corruption, disinformation, fcc, journalism, local news, media consolidation, propaganda, republican