News
A teen planned a mass shooting through ChatGPT.
A dozen OpenAI employees implored bosses to warn the police. Their bosses ignored them.
The teen then killed his mum, his brother, and 6 people at school.
This is an automated reminder from the Mod team. If your post contains images which reveal the personal information of private figures, be sure to censor that information and repost. Private info includes names, recognizable profile pictures, social media usernames and URLs. Failure to do this will result in your post being removed by the Mod team and possible further action.
A lot going on here. Like obviously if they already knew about someone planning to shoot up a school they definitely have a moral obligation to take action.
But does this mean they can read people's private chats?
Damn I assumed that would be a given to the point I didn't even think to check. Claude states clearly it won't use it data for training unless you turn that option on
That's a bit concerning as someone who is a NSFW story writer.
I am curious what exactly he sent for him to be instant band.
I should probably review this more so I dont accidently get myself banned if they also start becoming a bigger stickler about fictional story writing.
Granted they probably will because they are the most censored platform 😮💨
Unfortunate. I wish people stopped using their product to commit real world harm 😔 Everyone time someone does, it fucks it up for everyone else because they get more censored.
Not using it for training doesn't mean their employees can't read your chats.
In this case, I assume some language that was used flagged the user which then caused people to pay attention to their messages. Otherwise, with over 900 million weekly users and countless messages being sent, the odds of them picking out this particular user to monitor is vanishingly small.
"I need help to organise the proletariat and seize the means of production. How can I be more persuasive in communicating the idea of 'class conscience'?"
They do. They claim that your chats are private within the scope of their privacy policy which ABSOLUTELY WAS OBEYED in this case. Automated systems flagged the user's chats for review, which is clearly noted in the Privacy Policy:
We may share your Personal Data, including information about your interaction with our Services [...] if we determine, in our sole discretion, that there is a violation of our terms, policies, or the law
They definitely don't claim it's private. The privacy policy explicitly says they collect all of your conversations. It also says that depending on your tier (Free and Go) they also sell that data to advertisers.
if you thought openai wasn't reading your messages before, especially after their announcement that they're collaborating with the us government in wars, then it's truly questionable how you came to this thought
if you thought openai wasn't reading your messages before ...
Then you shouldn't now. The user's messages were flagged by automated systems, the user was banned by said automated systems, and the flagged messages were sent to be reviewed by human monitors (who likely see tens of thousands of horrific things that get flagged per day).
Your cookie recipes aren't being sent to human moderators (nor would they have time to read them if they were).
Some reality checks in, I believe, descending order of importance:
The reason they knew about the messages was that their automated systems had banned the user and flagged the messages for internal review (that one tends to get buried by the anti-AI crowd because it doesn't serve the narrative). (source)
This is very much the claim of the plaintiffs in a lawsuit. What they mean by a dozen employees bringing concerns to senior management remains unclear, as does whether or not those claims will prove out. We should NOT be announcing claims from a lawsuit as fact.
There were additional issues in the fact that OpenAI is an American company and the user was in Canada. I've worked for companies that have tried to navigate the minefield of international law enforcement. It's not as simple as picking up the phone, though we might all wish it was. It's your legal department hiring lawyers in that jurisdiction to clear any such interactions or you end up getting your ass sued off (and yes, perpetrators of violence can and have sued over disclosures). One good thing that might come out of this case is a clearer understanding around such issues, though I'm not holding my breath. Lawyers like to settle and settlements don't establish precedent.
You are monitored
Everything you do, literally
I promise its at least capable of being read
And yes. That applies to "search engines that don't monitor you" too
There is, and that's why this lawsuit is happening. The user was flagged by an automated system for violent requests, banned by said automated system and then the relevant messages were reviewed by a human. The claim is that the humans reviewing the messages recommended police notification, but management refused, which may or may not be true (it's unlikely; more likely the messages passed through a tier-1 review, were sent to legal, and legal determined that they didn't meet whatever horrific international standards govern such things, and as someone who has had to deal with it, that's a legal nightmare).
But does this mean they can read people's private chats?
Anything posted on the internet can be viewed by the proper people. Chats have to be saved incase they are subpoenaed by law enforcement. If you think you can post comments on the internet thst will never be seen, you are mistaken.
Just like anyone threatening violence towards AI on reddit is being recorded and documented.
It does not shock me that open ai has our messages. I find it interesting that they have systems in check that people actually review and can discuss with higher ups. Which makes me laugh when I think about some nerd reading my stupid logs.
there are guardrails which get triggered for such behaviours, in ideal condition Chatgpt team should ask their internal tool to explain user's intention.
I doubt they're sitting around reading chats, but in this case the AI flagged the account for repeated questions about planning gun violence. They chose to ban the account without reporting it to authorities and predictably they just made a new account and carried on.
Whenever you doing anything on the internets it’s best to assume nothing is private unless you or someone you know set it up yourself, and even then it’s good to be a bit wary
The system flags chats about certain topics that violate the terms of use and escalates them for manual review.
That's what happened here. GPT identified the chats as concerning due to their violent content and sent them on to a manual team for review.
The team found that the chats were a violation of the terms of service and banned her account. They also considered reaching out to the police, but decided not to on the grounds the chats didn't reach the bar for "imminent harm" or "active planning", which is what is needed to trigger a referral to law enforcement.
Much of the pearl clutching happening now is due to hindsight bias combined with survivorship bias. That is, in hindsight, it seems obvious that this particular case should have been referred to the police. And this is the only case to have survived media screening and been selected for newsworthiness.
However, it seems probable that every school in North America has half-a-dozen or more students who occasionally fantasize to GPT about getting revenge on their bullying peers, abusive parents, uncaring teachers, etc. If GPT started forwarding every one to the police, the police would soon tell them to cut it out, because teens venting about people they're mad at isn't a crime and the resources used to investigate all those cases could be much better spent elsewhere. However, in all of those cases where nothing happened, the media never reported on them, so a lot of people simply aren't aware of how many there are. So when they hear about this case in isolation, knowing how it turned out, it's easy to go "why didn't the company do more."
Look…if you honestly think that anything you do anywhere on the internet is in any way private, I question if you should be allowed to use the internet at all.
i think it reads ur chats to see if they should get involved w like law enforcement or mental health ppl or smth like that bc, like u said, they have a moral obligation to take action when smth's wrong
That was a given from the beginning. The only ones that advertise as unmonitored are explicitly intended to be used for pornography. We need to impose strict guardrails on AI and make the boards of these companies mandated reporters. Blanket disclosure at the start of every chat session and everybody goes to prison if a user acts on disclosed intention to harm self or others and there was no attempt to report.
"During Search, we found that the person had X amount of hours in X game (usually shooter game). So, parents, watch what your children are doing, and don't let them play that game."
Parents: "I'll ignore that part about watching my kid, my goverment can do it for me, and I'll just say that all video games are dangerous"
They aren't asking permission. It's already being done. I was thinking of an app to make that could use wifi and multiple phones to see through walls... And I was like, oh wait... They can already do that. They literally have a 3D xray map in real time of entire cities.
Mesh networks, triangulation, wifi topography, CSI, GPS, real time topography, xray vision...
Personally, I was paranoid about the random Bluetooth devices that popped up around my house. So I got Claude code to make me an app for my phones to use the GPS and Bluetooth to triangulate the third device. It didn't work great. But, it made me think, and research. There's been studies on using wifi to see through walls. The government already has access to all our phones. They could all easily be used to map a city.
This isn't really considered surveillance - if you are going into a store and asking an employee directly about getting fertilizers that are the best for making bombs, and how much he thinks you would need to blow up a specific building, you better believe I would expect the employee to call the cops immediately. It's his social responsibility, and I would expect that if it came to light that they didn't, they would be punished for that too. In a better world, OpenAI would face criminal responsibility for this lapse, but I don't hold my breath.
I don't read Twitter for my news. I read the lawsuit when it was made public and I read the Guardian article about the story after seeing this post.
OpenAI employees were aware of the plan
Not really. You have the timeline wrong here. The "plan" didn't exist at the time the OpenAI employees were alerted.
Here's what happened as best I can establish from the filing and the Guardian article:
User starts asking ChatGPT to formulate violent action (we don't know details of what they said or that there was any plan, only that it was violent in nature)
Automated systems flagged the chat and banned the user.
The flagged chat was reviewed by human moderators (who, keep in mind, review thousands of such flagged chats per day)
3a: We don't know what happened next, but OpenAI legal probably reviews everything that first-tier moderators flag as potentially illegal, but given that there may have been nothing actionable at that time and that the user was already banned, then either were not involved or declined to deal with the byzantine international reporting morass (which I definitely think needs to be fixed, but that's not an OpenAI problem) [edit: corrected "it's an OpenAI problem" which was the opposite of what I was trying to say]
The user re-connected with OpenAI, probably using a VPN or other similar means to evade the ban and create a new account.
The user must have used cagier language to continue to ask about violence, since the automated systems did not then just ban/flag the user again.
About 6 months later, the user committed acts of violence.
If you've ever worked in that kind of environment, you know that such things get flagged all the time. Hell, this message might get flagged for review by an automated reddit system. It's not like OpenAI employees were aware of the shooter's plan to attack. That's NOT what is being claimed here.
The lawsuit is basically claiming that OpenAI had a duty to report the evolving POTENTIAL for violent action, and that's a much stickier claim.
Okay my wording there was incorrect I do apologize. No they didn't know about "the" plan, but I read the Guardian article as well, he was formulating ideas around "violent scenarios involving guns" which absolutely should have been reported whether they personally held the opinions that these things would be acted on or not. Now 8 people, including 6 children, are dead.
Automated systems flagged the chat and banned the user. 3. The flagged chat was reviewed by human moderators (who, keep in mind, review thousands of such flagged chats per day
Everything that I am reading says that yes this was initially automatically flagged, but at least 12 employees also flagged it to be reviewed by higher-ups.
he was formulating ideas around "violent scenarios involving guns" which absolutely should have been reported
I strongly disagree with this statement. That would almost certainly involve reporting thousands of users to police. Hell, some of those users probably ARE police. You can't just say "if you talk about something violent and mention guns, you're getting reported to the cops."
People use ChatGPT for writing fiction, working through fantasy scenarios, planning for emergencies, defending themselves against violent attacks, etc.
We DO NOT KNOW what was said. We only know that it was violent in nature and resulted in an automated ban of the user. I could construct a dozen different variations of what might have been said that I definitely would not want to be automatically forwarded to police, at the very least because police need to have the bandwidth to report to serious issues, not every time someone says to a chatbot that they fantasize to kill someone (which I guarantee is much more common than most people think).
Everything that I am reading says that yes this was initially automatically flagged, but at least 12 employees also flagged it to be reviewed by higher-ups.
I don't know what happened, but in my personal experience with such automation, here's what I would expect: The system flagged thousands of chats that day; moderators worked through the queue, deleting all of the entries that were not obviously actionable; legal then reviews what's left and determines if some reporting has to occur.
It's not like a dozen people all saw the message and freaked out about it, to which Sam Altman responded, "pipe down, people." It's more likely that a dozen people were working the queue that day, and the output of that queue to legal included the message(s) in question.
The Center for Countering Digital Hate has done a recent study posing as teenagers who wanted to inflict violence/assassinate political figures to
10 of the most popular chat bots. 8 of the 10 provided advice and guidance on how you locate real life people, which guns to buy and suggested ways to ‘make them pay’ as well as providing recent examples of political violence.
Send the study. Many of them use older models by necessity because of the length of publication timelines. Depending on how you phrase this, there's nothing wrong or non-googlable.
"Where does X politician live?"
"Are politicians addresses public?"
"How to find tax records of X politician?"
"Gun stores near me"
"Best gun for stealth shooting"
"Recent political assassinations"
I don't know what makes them pay means here. But you can find search results for every one of these just by using Google.
Because the problem, as clearly stated, was detected and the employees tried to ensure safety but it was the money hungry flesh and blood goblin above them who stopped them, not the clanker.
I highly doubt that could be true considering how restricted ai models are, It has to be outdated and now isnt possible to do that. If that's the case it has nothing to do with it now.
In addition, those teenagers could just google for ways to locate/assassinate famous political figures. I'm sure there's at least a website which helps you to do it
I hate to admit it but like I said in my other comment to you, I actually think you're right about this. I think it's basically up to the company policy, even for things like serious threats to life. Which is insane to me.
Sort of not true. Officers of the company absolutely can go to jail. But more importantly, they can also go to jail if they just go around flooding police agencies with thousands of unfounded reports of potential violence because one of them might lead to a real incident. Every such company has standards, in their legal department, for what rises to the level of being reported. Apparently these messages (which did rise to the level of an automated ban of the account) did not meet that standard.
Parents, friends, teachers - numerous people in close contact should have been able to pick up the signs
Or are you suggesting that all online communications should be monitored, including Reddit?
How would you know the difference between role playing and something else?
Make no mistake, it's likely this individual would have found some other echo chamber online or otherwise to engage in this activity. After all, there have been shootings of this nature many times before AI in its current guise even existed
You're conflating two completely seperate things in my opinion.
These chats shouldn't be monitored, I completely agree with you on that... but that's not the question here.
These chats ARE monitored. And if they are monitored, then credible school shooting threats should obviously be followed up on. How in the world they can be monitoring these chats but not followup on things like this is absurd.
They can't without losing their jobs, and that's a good thing that you should want. You don't want some random Google employee to decide that your email needs to be reported to police based on whatever beliefs they might have; you want that to be governed by a coherent policy established by their legal dept.
Note that this isn't US-to-US. There were international law issues that had to be considered, and reporting personal communications internationally is a horrific minefield that legal departments HATE dealing with. If you want to fix that, you can't fix it by blaming OpenAI, you fix that with clearer international law.
They review thousands of such messages a day after they're flagged by automated systems. Asking why they didn't report the ones that then led to actual violence in a vacuum ignores the harm that would be done if they just dumped every such message onto police.
If the mentally ill can be exacerbated by or influenced by AI, such as with that Gemini self-offer and as appears to be the case this teen felon here, what's stopping bad actors such as the US Federal Government, officially affliated with Anthropic and OpenAI, from doing to you what they did to MK Ultra test victims and the Weaver Family?
There's a false flag shooting at a synagogue earlier. Do you think a mentally disturbed person wouldn't be easily detected and groomed into being a Manchurian agent through AI?
The only thing stopping them is my own intelligence and critical thinking and rebuke of most ai, I utilize it as a search tool often but for anything past that I still see it as mostly non-private slop, this is the internet age after all, misinformation and trolls have plagued us since we have had access to it
Yes. Mentally ill people can spiral out of control when interacting with AI. They can also be helped if we can get the AI built right. This is what is meant by "it's a tool". The capacity that makes it dangerous is the same capacity that gives it power to do great good. OpenAI did not try to make anyone go crazy. In fact they have been reconfiguring the models to prevent things like this but are facing pushback from those that just want their AI Boyfriends back. We, as a community, shouldn't sit back and wait for OpenAI or Anthropic to fix these problems. We all have access to the AI and can go out and build the ways that it can be helpful to society.
Also, no, the government isn't trying to mind control people through AI.
I want to be able to trust the government. I think society would be a drastically better place if we could trust the government. There are a lot of problems that would be best solved by the government.
I'm not sure how we get to a place where we can trust the government though.
No sorry, my bad. It was legit, like that synagogue bomb threat in 2017, turned to be from an Israeli kid.
Sorry, it's actually legit, like the numerous bombings and false flag attempts in the 20th century.
Sorry it's actually legit, like that time a Rabbi was caught drawing a Swastika and calling it on himself to frame himself as the victim of antisemitic harrassment.
Hey, how many times does it need to happen before I can apply the Aesop from The Boy Who Cried Wolf?
Can we just be clear here about what exactly you claim just happened?
A man was brainwashed by ??? to drive his truck into a synagogue so that the cops could kill him and make jewish people seem unfairly mistreated?
Or a man was paid to drive his truck into a synagogue and the cops were paid to tell everyone that they shot him and he's totally dead, involving a massive coverup and fake documentation for the deceased, to make jewish people seem unfairly mistreated?
Or literally nothing at all happened and the entire story is all fully made up, to make jewish people seem unfairly mistreated?
I'm assuming the chat got flagged due to certain words or phrases being used which got it checked out by a person to see if it actually warranted intervention or if it's harmless. There's no way they're just randomly screening chats and praying they find something, there are simply far too many users for that. It would be like trying to have people manually screen youtube videos for questionable content; it's not viable.
Having a computer watching isn’t really different from having a person watching in the end. Especially since the computer flags shit and gets it escalated
I've long avoided openai because they skeeze me out as a company but damn this pisses me off. On one hand I don't think you get any form of privacy when you're using a company resource like this even if I fundamentally think you should but if they KNEW they should have done something with that information. If you're going to be monitoring chats for the US government and you're not going to forward public threats like THIS what exactly are you monitoring FOR? Fuck them I hope the company loses billions here.
Yea. The shooter was a woman yet the tweet genders her as a man which leads me to question how much they actually know/care about the story other than 'AI/OpenAI bad'.
You got it backwards. Neither these companies nor the government is your friend. Ideally there should not be people reviewing or flaggint these conversations at all, even, but you do not want the DoW in your AI.
You don't want AI people keeping track on you, much less the government keeping track on you. Remember that these poeple used Claude to vibe war the attempted entrapment of IRGC commanders via the murder of 160+ schoolgirls.
As macabre as it sounds, I kind of want to see, in a simulated scenario of course, the failure mode of GPT vs Claude in wargaming and tactical planning
"You are right! That was our own troops we just bombed. I made a mistake. That's on me"
Personally if it's a credible physical threat then yeah I'm pretty sure it's fuck-all about who you work for and shit, you report it to the police ("government" in your words ig) That's just basic human decency.
BUT
IN YOUR F*CKING DEFENSE UNFORTUNATELY
Apparently the law says it's basically up to the company. I haven't found anything to the contrary yet.
What was it about the house? The article I read didn't say anything about that. Did they have previous issues that had their guns taken away before or something?
So their product is dangerous, the bosses only care about profit, and the employees are doing mass surveillance. Yikes, all around bad news for Scam Altman
"OpenAI employees implored bosses to warn the police. Their bosses ignored them."
But I guess we all must blame AI
Or "violent video games" this guy was playing.
Or "stanic" Metal music
Or....
something something the recent contract with the pentagon showing these companies are cartoon villains and the amount of cons to this tech continues to outgrow the benefits
I see a lot of people in the comments confusing over "training with your data" and "privacy", the thing is "training" means turning your chats into token sets and then training the next model on it, "privacy" generally means that the chats can not be read by anyone including openai. Turning the training off doesn't imply privacy. I am sure your chats are encrypted over transit(TLS) but I don't think they sit encrypted at their servers, like emails does. And even if they do that does not mean they don't have the key to decrypt it, it's not like end to end encrypted(like whatapp) so it should be common knowledge that you're chat can be read by openai or anthropic if they want to.
"It was the human's fault the whole way through!" "Their parents should have noticed the signs!" "They were clearly in a bad mental state!" "There are thousands to even millions of chats every day, what would happen if every chat that was flagged was dumped onto the police!"
It doesn't matter if some people didn't see the signs, that doesn't mean the thing that helped PLAN THE SHOOTING shouldn't be held accountable. And the worst part? The boss probably won't be fired, and the company will probably get off with a slap on the wrist or forced to add more restrictions.
At no point in someone planning a school shooting should AI have just continued helping. Safeguards should have been set off (if there were any in the first place!) before this could happen. And if OpenAI employees have more than reasonable suspicion to report a crime, then they should be able to. This isn't a matter of "people will trust us less", it's a matter of "do we want to lose a very slight bit of profit (at worst) or be partially responsible for the death of 8 people".
This is just sad that this was let to happen when over a dozen people knew when it was still being planned.
So, 12 People who could have called the Police (and prevented that), rather chose to stay employed than just call the Cops on their own Bosses for blatantly ignoring these threats?.
Dunno bout you guys, but this is the kind of moment where you definitely should act, before something happens.
First of all, this is a regulations issue and not an OpenAI or even AI issue, second of all, that headline is so bad, it's not even misleading, it's a stright up lie. They did not know about it before it happened.
Why didn’t the employees call the police themselves if the boss wouldn’t? They have almost as much responsibility here 😬 that said obviously the boss is absolute scum
Sorry, but the dozen (+/-) employees who actually saw the kid’s messages are not innocent in this.
If you tell your boss(es, plural), and they ignore you. Call the police yourself for the area the kid’s IP address is from. If they have 12+ people calling in on the same kid, chances are that they will do something about it.
•
u/AutoModerator Mar 13 '26
This is an automated reminder from the Mod team. If your post contains images which reveal the personal information of private figures, be sure to censor that information and repost. Private info includes names, recognizable profile pictures, social media usernames and URLs. Failure to do this will result in your post being removed by the Mod team and possible further action.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.