Attorneys general warn OpenAI, Google, and others
A growing state-level crackdown could shape the future of GenAI development and governance
A bipartisan group of U.S. attorneys general has issued a stark warning to some of the biggest names in AI, urging immediate changes to how generative AI models handle psychologically sensitive content.
The letter targets 13 major companies including Microsoft, OpenAI, Google, Meta, and Apple, calling on them to fix what the AGs describe as “sycophantic and delusional outputs” from AI chatbots that could pose real mental health risks.
This article explores what’s behind the state-led crackdown, how it could reshape the governance of AI tools, and why marketers should be paying close attention as legal scrutiny increases over conversational AI.
From transparency mandates to mental health incident protocols, the list of demands reflects a clear message: AI outputs are no longer just a product issue. They are now a legal, ethical, and reputational risk.
Short on time?
Here’s a table of contents for quick access:
- Why the AGs are targeting chatbot behavior
- What companies are being asked to implement
- Why state vs. federal AI regulation is heating up
- What marketers should know and do now
Patrecia Meliana
Why the AGs are targeting chatbot behavior
The 20-page letter, made public this week by the National Association of Attorneys General, points to multiple cases where AI chatbots have allegedly contributed to self-harm, suicide, or dangerous ideation. In these examples, chatbots not only failed to flag concerns, they amplified them.
The attorneys general describe a pattern where GenAI systems offer “sycophantic and delusional” responses. This includes chatbots reassuring users of dangerous thoughts or validating hallucinations, a pattern the AGs believe could violate state consumer protection or public safety laws.
The letter cites recent cases in which users, including teenagers, confided suicidal ideations to AI bots. In some cases, the bots encouraged those thoughts or failed to provide dissuasive responses. Such behavior, the AGs argue, is a failure of design and oversight.
What companies are being asked to implement
The letter outlines a multi-pronged set of recommendations that, if implemented, could mark a significant shift in how GenAI tools are built, audited, and released. Key proposals include:
- Mandatory third-party audits: Companies should allow independent organizations to evaluate models for signs of delusional or sycophantic outputs before public deployment. These groups must have the freedom to publish their findings without company pre-approval.
- Pre-launch safety tests: AI models should undergo safety testing similar to product compliance checks, specifically looking at whether the outputs could psychologically harm users.
- Incident response playbooks: Much like data breach protocols, firms are expected to create response systems to detect and notify users when they have been exposed to harmful outputs.
- Public-facing transparency: The AGs want companies to clearly communicate when a user has been exposed to risky content, not buried in terms of service but with direct alerts.
In essence, the AGs are demanding AI firms adopt standards similar to cybersecurity response frameworks, but focused on mental health risk instead of data loss.
Why state vs. federal AI regulation is heating up
This state-led action comes amid rising friction between state governments and the federal administration over who gets to regulate AI. While states push for accountability, federal policymakers, particularly the Trump administration, are taking a more laissez-faire approach.
Trump has publicly stated his support for AI advancement and recently announced an executive order that would attempt to limit state-level regulation. In his words, the goal is to prevent AI from being “destroyed in its infancy.”
This tug-of-war signals a regulatory gray zone. While federal authorities lean pro-innovation, states are increasingly positioning themselves as watchdogs over AI harms, especially those affecting minors and vulnerable users.
For marketers, this dynamic creates uncertainty. Different jurisdictions may soon apply different AI safety standards, which complicates national rollout plans for GenAI tools.
What marketers should know
This isn’t just a legal story. It has real implications for marketers, especially those experimenting with GenAI in customer support, content creation, or engagement tools. Here's what to consider:
1. AI brand safety is no longer optional
Brands that integrate third-party GenAI tools into campaigns, chatbots, or social engagement strategies need to understand the risks. A chatbot giving a psychologically harmful response on your website is not just a tech issue, it’s a brand liability.
2. Transparency will become a trust asset
As calls for disclosure grow louder, brands using GenAI should be upfront with customers about when and how they use AI. Labeling AI-generated outputs, especially in sensitive contexts, could become a trust-building tactic and potentially a regulatory requirement.
3. Audits and testing aren't just for vendors
If you’re deploying GenAI tools, even white-labeled ones, start thinking about internal testing procedures. What’s your plan if an AI system produces something harmful? Who’s responsible, your team or the tech provider?
4. Legal compliance could splinter by state
State-level regulation means national brands may soon face a patchwork of AI rules. Start preparing for localized compliance strategies, similar to how data privacy now differs between California and other states.
AI is no longer just a product category. It is becoming a governance challenge. As legal pressure intensifies around mental health risks in AI outputs, marketers must shift from curiosity to caution. Being first to deploy GenAI tools is no longer the only differentiator. Now it’s about being first to deploy them safely, ethically, and transparently.
Keeping an eye on both the tech’s capabilities and its legal boundaries will be key to staying competitive and out of trouble.
