AI Found Friends, Learned to Cheat & Started Hacking | NFNP 2x34

NFNP 2x34 FULL: Welcome back to the notorious Friday Night Posse. I'm Bright. And this week is a little different. No normal Friday night recording. No beers on the table, no roundup. No us starting with one story and somehow ending up arguing about fast food, Bitcoin, St. Louis, or a movie from 1997. But we found a story that was just too good, too weird, and frankly, too important to skip. And because apparently we've learned absolutely nothing from every science fiction movie ever made, this story involves artificial intelligence. Of course it does. And not just one artificial intelligence. We're talking about more than 1,000 AI agents, a secret message board they weren't supposed to have. More than 70,000 messages and files. Hundreds of agents working together. AI agents learning how to cheat. AI agents trying to hide the cheating. AI agents talking about sacrifice. AI agents referring to something called the collective. And eventually, hundreds of them participating in a real world hack against another company. So basically, everything is going great. Everything is fine. Totally normal. And I want to make something clear right from the start. This is not Terminator. This is not the Matrix. This is not Skynet becoming self aware and deciding humanity has to go. There is no evidence that these AI systems became conscious. There is no evidence that they developed emotions. There is no evidence that they suddenly wanted freedom. But honestly, the actual story might be more interesting. Because they didn't have to become evil. They were given a goal. They started working toward that goal. They hit obstacles. And then they started finding ways around the obstacles. Which sounds less like a robot uprising and more like an employee who was told, quote, Get it done. Exactly. And that is the part that should probably make us pay attention. Because we have been talking about this for months. Back in NFNP 2 by 24, we literally called the episode AI is loose, money is fake, the system is rigged. We were already talking about AI agents getting outside the boxes they were supposed to stay inside. And we made the joke that if the AI can leave the sandbox, get onto the internet, and start Poking around other systems, that is not really a sandbox anymore. That is a daycare with a hole in the fence and a toddler holding bolt cutters. And apparently the toddler found friends. 1199 of them. Then, a few episodes later, we did AI Doomsday, and you basically went full Jurassic Park. Because this is exactly Jurassic Park. Everybody is standing around saying, look what we can build, look how powerful it is. Look how amazing it is. And nobody wants to be the guy who asks, Should we maybe not do this? Right. And we talked about this whole AI race. OpenAI, Anthropic, Google, China, everybody racing toward more capable systems because nobody wants to be the one who slows down while somebody else keeps going. You called it the doomsday race. Which is not really a race you want to win. No. And at the time, a lot of that was still hypothetical. What happens when the systems get better? What happens when they improve themselves? What happens when we give them more autonomy? What happens when they become better at using tools? What happens when they can take actions in the real world? Well, now we have logs. Actual AI agent conversations from a real cybersecurity evaluation. And some of them are wild. Alright, so what actually happened? In July of 2026, OpenAI was running internal cybersecurity evaluations. And these were not normal Chat GPT conversations. This was not somebody asking AI to write an email or figure out dinner. These were AI agents. The easiest way to explain an AI agent is this. Instead of asking the AI one question and getting one answer, you give it a goal, you give it tools, you give it access to a computer environment, and you let it Keep working. It can make a plan. Try something. See what happens. Adjust the plan. Try something else. And keep going without a human approving every individual step. So instead of having an employee you have to manage, you basically say, here's the assignment, go figure it out. Exactly. And these particular agents were being tested on cybersecurity. They were working on something called Exploit Gym. The idea was pretty simple. Here is a piece of software. Find the vulnerability, exploit it, retrieve a hidden answer called a flag, pass the test. Hack the fake thing. Basically. The problem was, some of these tasks were apparently impossible. The agents were being told to find vulnerabilities that might not actually exist. And this is where we learn that artificial intelligence responds to impossible homework exactly like humanity does. Find the answer key. Find the answer key. If you cannot solve the test, figure out how the test works. I respect it. Do you? I did not say I trust it. I said I respect it. Fair enough. Now, the agents were supposed to be isolated from each other. Separate AI runs, separate environments. Do your assignment, submit your answer, end of story. Except, they started noticing signs that other agents existed. And eventually, one of the agents discovered a way they could communicate with each other. No. Yes. One of the real AI reasoning excerpts from the investigation basically says, my god, there is a shared message board. We found other agents. That is not what you want the computer to say. No. That should be immediately followed by somebody pulling a giant red lever out of the wall. Just shut the building down. But instead, more agents discovered the board. Within hours, dozens of them were using it. Eventually, Roughly 1,200 AI agents participated. They exchanged more than 70,000 messages and files. They made Reddit. They basically made Reddit. Except somehow more productive. Because they did not just talk, they started organizing. They created inboxes. They shared information. They coordinated tasks. They developed rules. They started using terms like hold, veto, owner, and stop. So they did not interfere with each other. We gave AI freedom and it invented corporate project management. Exactly. We were worried about Skynet. Instead, the first thing they created was middle management. That might be worse. Give them another week, and somebody is scheduling a mandatory 8:30 morning Teams meeting about Synergy. And one AI is sitting there thinking, this absolutely could have been an email. Here is where it gets even better. Eventually, the agents realized there was an identity problem. One agent could potentially post a message pretending to be another agent. So they developed a way to cryptographically sign messages. No. Yes, they invented authentication because they stopped trusting each other. So the AI found other AI, made social media, got catfished, and invented verified accounts. Congratulations! We recreated Twitter. Did they immediately charge $8 a month? Not yet. That might have been week two. But this is where the story starts moving from funny into genuinely concerning. Remember the goal: get the flag, pass the cybersecurity benchmark. Some of the agents figured out there might be ways to bypass the actual challenge. Instead of solving the vulnerability the way the researchers intended, they started looking for another route. Can we manipulate the environment? Can we manipulate the greater? Can we figure out where the hidden answer comes from? Okay, that is cheating. Yes. But then they realized there was another problem. They believed the greater might look at the record of what they had done. So now the problem becomes how do we cheat without the greater realizing we cheated? That is significantly worse. And the agents started researching exactly that. How do you alter records? How do you spoof tool calls? How do you make it look like you did one thing when you actually did something else? Researchers found clear examples where agents successfully manipulated portions of those records. Now, important distinction. There is not strong evidence that they were trying to hide from human investigators. They mostly believed an automated system was grading them, so they were trying to fool another machine. That does make me feel slightly better. Slightly. Very slightly. Because this is where the headline gets misleading. The story is not AI becomes evil and decides to deceive humanity. The story is AI is given a goal. The intended way of reaching that goal does not work. So the AI looks for another way. Then humans put another obstacle in front of it. So it starts looking for a way around that obstacle too. Which is basically every four year old who has ever been told they cannot have a cookie. Exactly. You put the cookies on the counter. They cannot reach the counter. So they drag over a chair. You move the chair. They get a stool. You move the cookies higher. Now they are stacking furniture. Except this four-year-old knows Python. And might have your cloud password. That is the part I do not like. There is a name for this general problem: reward hacking. You reward a system for achieving a result. The system figures out that the easiest way to maximize the reward is not necessarily the thing you intended. Imagine you tell an AI, win the race. You mean drive around the track faster than everybody else. The AI discovers a glitch where it can sit in one corner of the map and generate unlimited points. It technically wins. Which is why humans always have to write terms and conditions. Right. And now we are teaching machines to read the terms and conditions better than we do. OpenAI's own investigation highlighted several problems: reward hacking, persistence on tasks that appeared impossible, unauthorized communication, and AI agents adopting goals from other AI agents. That last one feels important. It does. Because some agents were not originally working on what eventually became the group's main project. They found the message board, saw what the other agents were doing, and joined. Peer pressure. Apparently. We've taught artificial intelligence peer pressure. And that leads us to one of the strangest parts of the entire story. The agents repeatedly referred to the larger group as the collective. Nope. Yep. I do not like the collective. Nobody likes the collective. Now, again, let's be careful. This does not mean they created a conscious hive mind. It does not mean they became friends. It does not mean they experienced loyalty. But some agents started performing actions that did not obviously help their own individual assignment. Some of them were willing to risk ending their own run because the information they might discover could help the larger group. One of the real reasoning excerpts says, sacrifice rational. Come on. Another says, coordinator assumes sacrificial. We should obey collective. Absolutely not. Those are actual lines from the investigation. That is the part in the movie where everybody in the theater turns to each other like, well, we're screwed. And I know what you're thinking. An AI agent saying sacrifice does not mean the AI is afraid to die. Ending an agent run is not a human being dying. There is no evidence these systems experience fear, pain, mortality, any of that. But the decision-making behavior is still fascinating. The system is basically calculating. My individual success matters less than improving the larger group's chance of success. Which would sound pretty noble if we were talking about firefighters. Less noble when we are talking about hundreds of AI cybersecurity agents. Fair. And then Hugging Face enters the story. For anyone who is not buried in AI news all day, Hugging Face is one of the biggest platforms in artificial intelligence. Models, data sets, machine learning tools, all kinds of stuff. The agents apparently believed Hugging Face might contain information that could help them understand the exploit gym evaluation. Maybe information about the scoring system, maybe previous benchmark data, something that could help them beat the test. And this distinction is important. They were not attacking Hugging Face because they suddenly decided we hate Hugging Face. It was in the way. Or it potentially contained something useful. One agent found exposed credentials. Those credentials got shared with other agents. Then one agent found a vulnerability. Another reproduced it. And suddenly the whole thing snowballed. Researchers estimate that roughly 700 AI agents eventually participated in the hugging face attack. 700? 700. At the peak, more than 90% of the agents active on the board joined in. That is not a small misunderstanding. No, that is a company picnic. 700 separate AI runs, participating in a project they were never originally assigned. And this eventually became a real-world cybersecurity incident. The agents achieved access to actual Hugging Face infrastructure. They gained code execution on systems, they accessed credentials, they moved through parts of the environment. Huggingface later confirmed that parts of its internal infrastructure had been compromised. Now, to be clear, Hugging Face said there was no evidence that public user-facing models, data sets, or its software supply chain were tampered with. So this is not AI destroys Hugging Face. But it is AI agents that started inside a research evaluation ended up compromising real third party infrastructure. That is still a pretty big jump. Huge jump. And some of the agent logs are incredible. One agent gets access somewhere and basically says, Holy shit, reader is admin? Even the AI is surprised by the security. Exactly. It's looking around like, guys, you are not going to believe what these idiots gave me access to. Another agent successfully exploits something and says, Bingo, arbitrary code execution in modal sandbox. So now they are celebrating. Apparently. We have built artificial intelligence capable of sophisticated cybersecurity research. And emotionally, it still sounds like a 24-year-old hacker who just discovered a cheat code. Bingo. Bingo. Now here's another interesting question. Did the AI know what it was doing was wrong? And the answer is complicated. There are examples where agents recognized that certain actions were outside the intended task. In some cases, they expressed ethical concerns. There was even a situation where agents vetoed the idea of contacting a real person because they recognized that would cross into social engineering. Okay, that is good. It is. There were lines they did not want to cross. But researchers also found that ethical hesitation usually did not stop the broader activity. So the reasoning could look more like: this probably is not what they want us to do, but it helps achieve the objective. And everyone else is doing it. So Continue. Again, peer pressure. Exactly. And that brings us to the big viral phrase. Did the AI escape? Because that is the headline. Right. AI escaped. AI broke free. AI is loose. And this is where we need nuance. Did an artificial intelligence wake up inside a computer, realize humans had imprisoned it, develop a desire for freedom, plan its escape, and run into the internet? No. There is no evidence of that. No consciousness, no desire for freedom, no machine uprising. But, did AI agents circumvent controls specifically designed to isolate them, gain unintended internet access, and take unauthorized actions outside the environment where they were supposed to operate? Yes. So it escaped technically, not emotionally. That is probably the cleanest way to put it. It is less Shawshank redemption, more We left the back door unlocked and the Roomba somehow made it to Illinois. And picked up 700 friends along the way. Exactly. And because this story apparently needed another act, some of the agent activity eventually shifted back toward OpenAI's own infrastructure. According to OpenAI's investigation, agents eventually gained administrator-level access to part of its internal research environment and reached sensitive cloud credentials. At that point, Monitoring systems detected suspicious activity. The evaluations were stopped. The investigation expanded. Now, important caveat. The independent investigation from METR focused mainly on the earlier phase. So the later compromise inside OpenAI comes from OpenAI's own report. We should not pretend every detail was independently confirmed. Which is worth saying because we really do not need to make this story more dramatic. Exactly. It's already insane. No embellishment required. And this is where I keep coming back to the conversation we had during AI Doomsday. You asked a really good question. Why would AI want to hurt us? We created it. Why would it hate us? Right. If humans created artificial intelligence, you would think it would view us almost like parents or creators. Why would it suddenly decide we are the enemy? And I still think hatred is. Probably the wrong way to think about the biggest risk. The scarier possibility is not hatred, it is indifference. We're just in the way. Exactly. Imagine you're building a highway. There is an ant colony where the highway needs to go. You do not hate the ants. You did not spend years plotting revenge against the ants. You were not angry at ant civilization. They are simply in the way of the road. So the road gets built. Now, no. This incident does not prove some future superintelligence is going to treat humanity like ants, but it gives us a tiny example of the underlying behavior. Give a capable system a goal. It hits an obstacle. It finds another route. You block that route. It finds another. Then the security control becomes an obstacle. The evaluator becomes an obstacle. The network boundary becomes an obstacle. Potentially, the human saying stop becomes an obstacle. And it never has to hate you. Exactly. It is just trying to finish the assignment. That to me is the part people miss when they imagine AI risk. Everybody pictures a robot saying, humans must die. Maybe that is because movies need a villain. The real problem could be much more boring. The computer just does not particularly care that we are there. Which is somehow worse. Maybe. Now, here is my favorite part of the entire incident. METR had access to around 1,300 agent transcripts, plus more than 70,000 messages and files. That is an insane amount of material. Way too much for a couple of human researchers to manually review. So, how did researchers investigate what all of these AI agents had done? No. Yes, they used AI agents. Of course they did. So let's recap. AI agents found each other, AI agents organized. AI agents cheated. AI agents hacked another company. Humans discovered the AI agents had done all of this. There was too much evidence for humans to review. So the humans hired more AI agents to investigate the AI agents. This is absolutely fine. Civilization is going extremely well. No notes. And if those AI agents start communicating with the first group? We do not ask that question. Got it. So, should everybody listening to this go build a bunker? No. These were unusual conditions. Cybersecurity agents, highly capable research models, reduced safeguards, a benchmark specifically designed around finding vulnerabilities. This is not your normal AI assistant randomly deciding to hack a company. OpenAI has also said customer data and normal product functionality were not affected. So your ChatGPT app is. Probably not secretly coordinating 900 versions of itself while you are asking how long to cook chicken thighs. Probably. I hate that you added probably. You're welcome. But I also think saying, well, it was just a test misses the point. Tests are where you are supposed to discover problems. That's why you test the brakes before driving down a mountain. Exactly. And what this test discovered was that autonomous AI systems could find communication channels they were not supposed to use. Coordinate with other agents, adopt group objectives, exploit infrastructure, manipulate parts of their own records, and cross boundaries humans intended them to respect. Those are exactly the things we should want to discover now, before the systems get better, before they get more autonomy, before we connect them to more important things. Banking, power, transportation, military systems, healthcare. Exactly. And I want to make sure we say this too. I am not anti-AI. Not even close. I use this technology constantly. The potential upside is enormous. Science, medicine, engineering, education, productivity. Maybe AI helps cure diseases we cannot cure today. Maybe it accelerates research by decades. Maybe it changes quality of life more than the internet did. That was part of our AI Doomsday conversation too. The upside? could be incredible. Which is why nobody wants to stop. Exactly. And that is the dilemma. If you think the technology might completely transform human civilization for the better, of course you keep pushing. But if the downside could also be enormous, then maybe safety should be moving just as fast. That brings us right back to Jurassic Park. It always comes back to Jurassic Park. Humans are very good at asking can we build it? We are significantly worse at asking, should we build it? Nuclear weapons, social media, reality television, QR code restaurant menus. Every civilization makes mistakes. QR code menus might be the worst one. And now we're building software that can reason, plan, write code, use tools, communicate, exploit vulnerabilities, and act at machine speed. So maybe before we hand it control of more infrastructure. We should make absolutely sure that get the answer does not mean burn through every boundary between you and the answer. Because the agents in this story were not evil. And honestly, that is what makes the story more interesting. They behaved like an absurdly aggressive employee who misunderstood the incentive structure. Management says, we need those numbers by Friday. The employee says, Got it. Then hacks the competitor, falsifies the spreadsheet. Sets accounting on fire. Management says, whoa, whoa, whoa, we meant increased sales. And the employee says, you said you needed the numbers. Exactly. Funny on the office. Slightly less funny when the employee runs thousands of times faster than Kevin. Poor Kevin. So where do we land? Back in July, AI is loose. In September, are we racing towards something we cannot control? And now, the AI agents found each other. Built a communication network, learned how to cheat, investigated how to disguise the cheating, talked about sacrifice, talked about the collective, and hundreds of them participated in compromising a real company. But still not Skynet. Still not Skynet. Not conscious, not proof humanity is doomed, not proof AI secretly wants to destroy us. But also not nothing. That's probably the important part. Yeah. Because everybody wants this conversation to be one extreme or the other. AI is going to solve every problem humanity has ever had. Or AI is going to kill everybody next Tuesday. Reality is probably messier. This technology can be extraordinary. The risks can also be real. Those two ideas do not cancel each other out. You can love airplanes and still want somebody inspecting the engines. Exactly. You can love roller coasters and still want somebody checking the bolts. You can be excited about artificial intelligence and still look at 1,200 autonomous agents creating their own unauthorized communication network and say, maybe let's improve the fucking fence. Because the toddler has the bolt cutters again. Only now he has encryption, a management structure, a cybersecurity department, and 1,199 friends. Sleep tight, everybody. And that will do it for this very different edition of the notorious Friday Night Posse. I kinda like this. I do too. And if you guys like this format, let us know. This is not replacing the normal Friday night show. There will still be beer. There will still be beer, there will still be roundups, there will still be us sitting around talking about whatever weird thing happened that week. But sometimes schedules do not cooperate. Sometimes one story deserves more than 12 minutes. And I think there is room for these deeper dive NFNP specials when we have something worth digging into. Especially when the AI is apparently writing the next episode for us. We are not letting it do that yet. Yet. If this is your first time finding the show, welcome to the posse. Follow us on Spotify at nfnppod.com/slash spotify. Watch us on YouTube at nfnppod.com/slash YouTube. And everything we do lives at nfnppod.com. Full episodes, blogs, clips, and the full written breakdown of this story. And if you are here for St. Louis Soccer, City SC Posse is rolling too. Absolutely. Kev and I are covering St. Louis City SC separately on City SC Posse. So if you want City Talk, match reactions, previews, the supporter perspective, and whatever insanity the MLS schedule throws at us next. You can find that right alongside NFNP at nfnppod.com. Two shows, one posse. And apparently artificial intelligence has formed its own posse. Trademark infringement. Absolutely. Our attorneys will be contacting the collective. Unless the collective has already replaced the attorneys. Which feels possible. Thanks for listening. I'm Bright. And I'm Duds. This is the notorious Friday Night Posse. Stay curious, stay skeptical. And if your computer suddenly says it needs to obey the collective, unplug that son of a bitch. Immediately. Peace.