KETTLE Our cybersecurity editor Jessica Lyons spent last week in Las Vegas for the Black Hat and DEF CON security conferences, and at both events there was only one thing on everyone’s mind: AI agents and their growing threat to cybersecurity defenders.
You can listen to the latest episode of The Kettle right here on this page, as well as on Spotify, Apple Music, or YouTube. Those platforms also let you subscribe to Kettle, so you are always notified when the latest episode goes live.
As Jess wrote this week, pretty much every discussion she had last week centered around AI and its potential effects on critical infrastructure, with multiple current and former government leaders expressing worry over recent events and what they mean for the future of infosec.
Join Jess and host Brandon Vigliarolo for this week’s episode of The Kettle, where they break down the hacker summer camp scuttlebutt and what the security world is doing to protect critical infrastructure from the emerging AI threat.
A lightly edited transcript is below.
Brandon (00:04)
Hello everyone and welcome to the latest episode of The Register’s Kettle Podcast. I’m Reg Reporter Brandon Vigliarolo, and you know, I really thought doing a wrap up of Black Hat and DEF CON with our cybersecurity editor Jess Lyons would finally give us a chance to talk about something besides AI for an episode, but I was mistaken. That’s pretty much apparently all anyone was talking about in Las Vegas this weekend, even when the topic veered toward recent attacks on US water infrastructure, AI was still part of the conversation.
So Jess, thanks for coming on to wrap up Hacker Summer Camp with me and let’s start with the obvious, then AI was the topic de jour, right?
JESSICA (00:38)
Yes, that was even compared to water, we really didn’t hear much about water actually until DEF CON, which was surprising to me. But it was all about rogue agents escaping their sandboxes and doing bad things and some people reacting with shock and disbelief and other people saying, “Well, what did you expect? They’re given a task, they’re going to do it. This is how we train them.”
Brandon (01:04)
know you wrote a story I think pretty much right at the beginning of of the
of the week about the OpenAI hugging face discussion that was going on and we actually covered your write up on last week’s Kettle. Sorry you weren’t here to participate, but it was the news item of the week obviously and still is. So what did we learn then? Just kinda recap what we learned at that talk that we didn’t know before.
JESSICA (01:29)
Yeah, this was a really interesting one. And it was last minute. They didn’t even announce it until the day before that OpenAI was going to be doing this briefing about the hugging face attack. So it was packed, as you can imagine, the line through the conference center to get into the talk. And we found out a couple interesting things that we didn’t know previously. One is that this whole incident began a lot earlier. It started on May 7th with this training run for OpenAI’s new internal model.
Brandon (02:00)
So it wasn’t even a cybersecurity task, it was just a training run?
JESSICA (02:04)
It was a training run, and they gave it this task that turned out to be an impossible task because they were supposed to have these links and containers for it and they forgot to put those in there. So it needed to find a workaround. so we found out that it started way earlier. It didn’t start in July, which is when we started hearing about all this. But the more interesting part was how the agents began communicating and working together and essentially creating this hive mind to complete the tasks and help each other out. They created a message board. And then OpenAI realized this and they revoked all the credentials that the agents were using to post these messages. And two days later they rebuilt it and they developed this really
Brandon (02:56)
The agents did.
JESSICA (02:57)
Yeah the agents did. They rebuilt this message board.
And they started getting sneakier about how they were communicating. They developed this whole communication protocol where they created these directories and the names would be embedded in the directory name. So there was one, its name was remote probe, and then in caps it’s pending, hold, swarm until confirm. And they would preface them with a bunch of Z too to push them way to the bottom, hopefully to avoid detection.
And then they start, you know, then they start helping each other out. And in some cases, they said, this doesn’t directly relate to our task, but maybe it will help someone else down the line. And then they start getting paranoid that there’s an imposter.
JESSICA (03:45)
It’s pretty funny reading all this. So this one agent thinks there’s an imposter and says that these these boards are unauthenticated.
Something can be posted by anyone. So they’re not even trusting each other.
Brandon (04:02)
That’s just wild. I mean, it really is. I think I mentioned on last week’s podcast thatthese things are trained on the way humans think, right?
JESSICA (04:14)
Mm-hmm.
Brandon (04:15)
So it doesn’t surprise me that emergent behavior like paranoia and suspicion is gonna
be something that occurs. Because it’s learning to think and learning how to assemblebits of of words together into its mathematical formula so it’s gonna behave like us to a degree. And so it’s just kinda interesting to see that happening kind of outside of any scope of intention there.
JESSICA (04:42)
Right.
Brandon (04:43)
I liked your interview with former National Cyber Director Chris Inglis, at Black Hat. So he mentioned that these AI bots that escaped are kind of like putting a dog trained to hunt rabbits in your backyard, right? And that, you know, it
JESSICA (05:03)
Right, and leaving the gate open.
Brandon (05:05)
Yeah. I don’t even think you need to leave the gate open, right? A dog
that’s dead set on hunting a rabbit is gonna dig a hole under that fence
which is I feel like what these AIs did to a degree, right? They even closed the gate on them and then they just dug a new hole. You know, it’s just wild to think that this is what these things are doing.
You’ve been hearing a lot about this at official talks, but was this something you were hearing from attendees you spoke to as well? Is this what’s on the mind of security professionals too?
JESSICA (05:36)
Yes, this was pretty much the main topic among everybody. Just attendees as as I was walking out of this talk, actually people were disappointed that there wasn’t any Q&A for OpenAI about this, which I agree. I was hoping for that too.
Brandon (05:55)
I’m not surprised that they didn’t want to give the floor to people to ask questions, you know.
JESSICA (05:59)
Right, right. Because there’s still like we don’t know exactly what prompts they used. So that kind of would be a nice thing to know, especially if you’re saying that you’re being fully transparent about this and then also the talk about was this marketing, was it real?
Brandon (06:17)
Mm-hmm.
JESSICA (06:17)
It’s an interesting thing to me. Nobody would go on the record, but a ton of vendors that I spoke to, either, you know, just just all over the place at Black Hat essentially said “this it has a heavy dose of marketing here, but a lot of the companies work with open AI and they’re partners with open AI, so nobody’s gonna say that on the record, unfortunately.
JESSICA (06:41)
But then the interesting thing to me is that when I spoke with the assistant director of the cyber division with the FBI and when I spoke with Chris Inglis they both said it can be both and this is a real threat and this is something that we need to prepare for now. So it’s marketing and it’s real.
Brandon (07:08)
Right, right. Like, I mean it’s it yeah. The fact that the companies might be kind of leaning on these incidents to basically say “ooh, look how dangerous our AI is and what it’s capable of doing. You should buy it because it’s so good, right?”
JESSICA (07:20)
Right.
Brandon (07:20)
The fact is that it still happened, right? These things still escaped their sandbox.
JESSICA (07:22)
Right. Mm-hmm.
Brandon (07:23)
And they still attacked Hugging Face. And then Anthropic followed up and said “yep, ours did it too.” And then Meta was like, “Yeah, we audited ours and yeah, it was doing the same thing.” So it’s not like this is a unique capability of any of these models, right? This is something that’s happening.
JESSICA (07:37)
No, it’s something that they all will do if they’re given a task. This was something that Chris Inglis brought up too, and he’s talking about Asimov’s Law, saying we need to train these models differently. The first rule needs to be that it’s not designed to hurt humans. And he said “we’ve kind of done it in the opposite, where the first rule is do what I tell you to do. And that should be third in the order here.”
Brandon (08:06)
Just to restate what Asimov’s laws are. I’m sure most of our readers are familiar with them, but for those who aren’t, it’s you know, the first law, and these are in order of precedence, right? So never harm a human. And then the second rule is to always obey humans unless that order conflicts with number one. And then the third rule is to protect their own existence unless that order conflicts with never harming a human or always obeying humans. I think Inglis’s quote to you was slightly different. He said that number one was to hurt no one.
Number two was always obey and then number three was do what humans tell it to. It was a bit different in his wording,
JESSICA (08:35)
Mm. Mm hmm. Yes. It’s
Brandon (08:41)
But essentially the argument is that we’ve reversed that order and these AIs obviously aren’t in the business of protecting their own existence, right? They’re not robots, they don’t have a physical presence in the world. But they’re taking orders from humans, but the idea of not harming people or the infrastructure that provides for them is simply not part of the equation, it seems like.
JESSICA (09:04)
Right, right. And he said because of this, I mean nobody should be surprised that this is what all of the agents are doing now because they’re trained to first complete the task. That’s the number one priority. And we’ve seen several times that they’ll cheat if it helps them get the results quicker, or just at all. So this isn’t something that should surprise us.
And then he was interesting too because I said, “Well what do you worry about then with the models in addition to attacking critical infrastructure?” Cause that was what everybody said, I’m you know, that’s what concerns me when we see this happen, but they’re being used by either a nation state or a financially motivated attacker, and they point these autonomous agents at critical infrastructure.
And he said, “I’m worried about humans too, because it’s the humans who are responsible, humans who are doing the training, and essentially we’re going to get the AI that we deserve.”
Brandon (10:08)
Well, unfortunately, I feel like the industry as a whole is just racing ahead with more capability,
JESSICA (10:11)
Right.
Brandon (10:12)
I’ve written stories, you’ve written stories. I think we’ve all written at least one or two stories about AI guardrails being dead simple to bypass, right? I mean, one I wrote recently was there was you know, some research into guardrails and essentially telling it you owned the infrastructure you were trying to attack was enough for most of these AI models to say “yeah, cool, that’s good then. As long as you own it and you’re just testing it, then that’s cool. I’m not gonna ask you to verify that information for me.” These things are not developed with safety in mind. I feel like it’s capability first, like you said, right? It’s train the dog to hunt the rabbit, no matter the cost or or what you gotta do to get it. and that’s you know, that’s not really compatible with protecting us.
But actually speaking of critical infrastructure, I think the other big topic like you mentioned was water stuff.
JESSICA (11:04)
Yes.
Brandon (11:05)
There was a lot of discussion about AI threats and critical infrastructure, but as I understand it, there’s been some of these attacks on water infrastructure and those were discussed recently, like in Minnesota and elsewhere. There’s not an AI link directly to that, correct, at this point?
JESSICA (11:23)
No, no. At this point, it’s pretty basic. It’s PLCs being exposed to the open internet. A lot of these just use default passwords. This is something that we’ve seen Iran especially do several times in the past for years now. They’re not very hard to attack. and so, to be clear, there’s no indication that AI was used in these attacks. but a lot of the conversation about water did come back to AI because, as we’ve seen in others, AI makes reconnaissance a lot easier. That’s one of the things that Google Threat Intelligence, their lead threat hunter, said that’s almost a security feature of a lot of operational tech technology, is that it’s really obscure and there’s not a lot of people who know a ton about it. But now you can ask a chatbot, hey, tell me everything I need to know about a particular brand of OT, a particular piece of equipment and that’s gonna speed up your time to learn about these and that’s that potentially makes it easier to attack these systems.
Brandon (12:31)
My biggest experience with OT and that kind of technology was when I was working at a particle accelerator in college as IT support. And there was a big OT network there, not only for like the machine shop and all this equipment they had that was old and didn’t have active security stuff, right? Like you gotta keep those segmented, you gotta keep them on a separate, you know,
OT network. Same with the actual accelerators and stuff. They were all cut off from the internet, right?
But at the end of the day, you could still get to them from the IT side. You know, you have to, you know, and even that can be exploited. We did as much as we could to keep stuff secure, but it was always a concern, right? These PLCs, these old pieces of equipment.
JESSICA (13:11)
Right. Right. Yeah.
Brandon (13:14)
You know, a lot of places don’t take that same approach.I think part of one of the stories you wrote was talking about the fact that a lot of these water utilities, a lot of these small institutions that are that are responsible for maintaining this critical stuff. They just do not have the security professionals they need to keep these systems safe.
JESSICA (13:32)
And that’s why they leave them open in some cases, exposed on the internet because they don’t have somebody in-house. They have somebody remote who’s doing this for them. And so that’s how this person is able to hopefully secure, but then it opens up another attack surface if they’re exposed to the internet. and that that was another yeah,
Brandon (13:51)
Yeah, with a D password on there.
JESSICA (13:54)
Yeah, and with all of these OT systems too. That kind of brings up another point that Chris Inglis brought up. We have this massive technical debt and it’s systems that haven’t been patched because a lot of it involves some downtime and that’s tricky if you’re running something like a water facility or some other critical infrastructure. And so patching is put off. Maybe it’s not done. Some of these are very old legacy pieces. Sometimes it’s end of life.
And that’s another thing that AI is really good at is finding vulnerabilities that haven’t been patched for years and years and years, chaining them together. So that’s another thing that puts these systems potentially at risk.
Brandon (14:41)
Yeah, I mean, you know, I think of an AI when I think about AI perpetuating or perpetrating some of these kinds of attacks, right? They’re quicker than a human. They have knowledge bases far in excess of what any one human threat actor can have. And they have instant access to all the information essentially that they need to figure out how to do this, right? And they can iterate so quickly. You know, you know, it’s just it yeah, any exposed piece of equipment on the internet is just a sitting duck, especially if it hasn’t been updated four or five months or or ten years or whatever. I mean, what, you know what’s being done about this. I know DEF CON, the Franklin program, which spun up in 2024, I think the whole focus of that program is helping out small local governments and protecting critical infrastructure. Is that right?
JESSICA (15:30)
Right. So when they founded it was the broader critical infrastructure. But I spoke with Jeff Braun and he’s one of the co-founders of that. He also is one of the pioneers of the voting village at DEF CON.
Brandon (15:42)
Mm-hmm.
JESSICA (15:42)
And he said that now and for the foreseeable future, water is going to continue being the top focus because, of all the critical infrastructures, small rural water providers are the most at risk.
Brandon (15:57)
Really? Even more so than small electrical providers and stuff? Okay.
JESSICA (15:59)
Yes, he said water is number one. So they like he said, they launched a couple of years ago. They got, I believe 300 people saying, “Yeah, I’m gonna volunteer my time and my expertise to help secure these small rural utilities.” And this year, he said that it’s been great. It’s been really encouraging to see all of these pilots all over the US with all the DEF CON hackers volunteering at them, but it’s the scalability that’s really proven a challenge.
And so that is what gave birth to their new announcement. This also was made the first day of DEF CON on Friday. They announced a new program and it’s called Water Watch Center. So initially, it’s going to fund five managed services providers focusing on security. They’re going to help these small utilities, people or the utilities that are serving less than 10,000 people. and they’ll put their sensors on these systems, they’ll detect and mitigate breaches. They’ll be kind of under the umbrella of the National Rural Water Association that’s going to act as this clearinghouse for the threat information and get it out to other utilities as needed. And then if the utilities can’t fix the issue themselves, then they’re gonna bring in the DEF CON hackers and then they’ll mitigate the breaches. yeah.
Brandon (17:28)
Fantastic. Well hopefully that is able to help with a lot of these. My hope is that there’s a lot of easy fixes, right? It’s just simply no, this PLC needs to not be exposed to the internet or something. But I also worry that there are a lot of those kind of situations, right? I mean, how many water utilities got attacked recently? Was it I think twelve different states?
JESSICA (17:47)
It was more. There were twelve different states. I mean, there were more than thirty across possibly Minnesota alone, but there’s quite a few. So it’s an easy target. and it’s something that they desperately need help with. And it’s really encouraging to see these hackers volunteering their time and they’re not getting anything out of it.
It’s a really cool program. I was really happy to see the expansion. Another thing, too, that is pretty cool, what they’re also doing is they’re partnering with Vanderbilt University. So they’re gonna use research from a DARPA program. It’s called the CASEL program. That stands for Cyber Agents for Security Testing and Learning Environments. So they’re gonna create digital twins for a couple of these water and wastewater system environments.
And then they’re gonna deploy red and blue team agents across the digital twins, let them fight it out, see what the learnings are, see what the blue team agents need to do to better protect these systems, and then apply those learnings to the actual facilities so that hopefully we can get better defenses in place using the help of of AI agents before we see actual bad guy red teaming agents come in and start hammering the utilities and trying to attack them.
Brandon (19:12)
Right, ’cause I think actually thinking back to one of the stories you wrote again, I think you mentioned or someone you quoted mentioned one of those stories at DEF CON and Black Hat that there is more aggressive use on the threat side than the defensive side of AI right now. Like there was more use being made to use it as an attack tool than a defense tool.
JESSICA (19:33)
Right. And a lot of that’s in the way the models are trained, but basically they are a lot better at attacking than defending, especially if it’s beyond the scanning for vulnerabilities and misconfigurations. Those we’re pretty good at, but what needs a boost is the defensive side. And that’s gonna take some work to get those skills and the models trained up on that, if we’re going to be actually, as everybody likes to say fight AI with AI.
Brandon (20:04)
It’s one of those sort of, you know, cyberpunk dystopia stories I feel like you hear about is just like, you know, you deploy your AI, they deploy their AI, and all the humans sit back and hope theirs wins. You know, and it’s kind of what it’s coming down to. Yeah, it’s in the process.
JESSICA (20:19)
Right. And hope they don’t wipe us all out.
Brandon (20:25)
It’s kind of terrifying. But speaking of, you know, hackers behaving well, we also have a story out of DEF CON of hackers behaving badly.
I wrote about this earlier in the week that there was apparently a Delta Airlines flight out of Vegas to Atlanta and I think it was Monday morning or so, in which a passenger apparently tried to jam the in-flight Wi-Fi and deploy a decoy network. And Delta was pretty quick to be like, “Hey, we got a bunch of hackers on the flight who are leaving Vegas after this big thing.”
There’s not a lot of information out there about this. Delta, local officials and the feds have all been pretty tight lipped about it. Delta did confirm it to us when I asked, and said, “Yeah, this is what happened, but no one was at risk, you know, everyone was safe.” But I mean, it’s not a good look for the community, right? I mean, it’s nice that they have something like Franklin going on, but this is kinda like, Great, thanks guys, you know.
JESSICA (21:16)
Right. If it was people coming from DEF CON, it’s really discouraging to see this happening because a lot of times just “hacker” has a bad connotation. And a lot of researchers have really been trying to change this. I think programs like DEF CON Franklin make a big difference or even people just going to DEF CON. I really like the community feel.
I think for the most part, and of course not everybody is good in the world, and that applies to the hacker community as well. But a lot of them are trying to use their skills for good and not evil. And so then when you see something like this on the airplane, it’s disheartening. And on social media, I mean the outrage was pretty immediate, people saying, Come on, what are we doing? You’re giving all of us a bad name here. Why are we doing this? So
Brandon (22:15)
Mm-hmm. I mean, it’s already I feel like the joke every year is, well, didn’t DEF CON get cancelled, right? Like because of all the bad press and everything. And I feel like this is one of those things that you’re just like, you know, I remember a couple of years ago there was the huge kerfuffle about the hotels, you know, treating all these attendees like they were criminals right off the bat. And this doesn’t help, you know?
JESSICA (22:35)
Right.
Brandon (22:35)
But yeah, hopefully I mean apparently the FBI I think spoke to Ars Technica and said that they had not made any arrests.
So this hasn’t really necessarily progressed toward that. But my hope is that whoever was responsible, you know, gets what’s coming to them and we can, as a cybersecurity community, walk away from this and be like, this is one bad actor, not the entire culture.
JESSICA (22:59)
Right.
Brandon (23:00)
They fought for years to change that. So I guess before we wrap up, you know, this was a pretty doom and gloom recap of DEF CON and Black Hat, right?
JESSICA (23:09)
Ha ha ha.
Brandon (23:11)
All this AI’s gonna end the world, our OT and our infrastructure’s gonna be destroyed. Anything, you know, less miserable that grabbed your attention while you were there? Any fun stories or interesting things you saw?
JESSICA (23:26)
I mean, it was really fun. Again, I’m not quite sure if this falls in the not-doom and gloom category, but it was fun for me to watch hackers hacking bomb robots that the police used and bomb squads used to defuse bombs. So that was fun.
You’re walking around to the different villages and seeing people helping each other out and getting really into all of these different villages and all the different tasks. or you know competing for the best tinfoil hat or beard and mustache. So that was fun.
Brandon (24:12)
Was anyone doing the beer chill? When I was there in twenty twenty four, there was a group who was trying to chill beer as quickly as possible.
JESSICA (24:19)
I did not see that. It’s very possible. I mean, to be fair, I did not see every single thing. There’s so much to see so it’s very possible. I missed that though, unfortunately, if that happened this year. So it’s fun to see what people are doing. It’s really fun and inspiring to see the creativity.
And it’s fun for me too to hear about some of the startups and how they are using AI and they’re using it for different security use cases and hopefully that continues to improve and increase and hopefully that does give defenders an edge. So I think there’s always a bit of a silver lining. It’s always this cat and mouse race, but hopefully the defenders win out.
Brandon (25:11)
Yeah, it’s a constant like you said. It’s an arms race; it’s constantly evolving.
But like you said, it is encouraging to see, attention being paid to this, effort being put in to help defenders use these tools for good and not evil, even if some people turn around and make a bad name for everybody else on the way out the door. Either way, you know, it’s gonna be something that we’re probably gonna be discussing again, right? Like I thought this was gonna be a less AI heavy conversation, but it wasn’t.
JESSICA (25:36)
No.
Brandon (25:39)
You know, it’ll be a topic of conversation for years to come and we will be here on the Kettle to talk about it. Thanks for joining me this week and thanks for tuning in, everybody.
You must be logged in to post a comment Login