The AI bubble could expose your deepest secrets

1 hour ago

Thomas Germain

News image — (Credit: BBC/ Serenity Strull/ Getty Images)

BBC/ Serenity Strull/ Getty Images The hands of two people exchanging wads of cash for a phone with a password lock (Credit: BBC/ Serenity Strull/ Getty Images) — (Credit: BBC/ Serenity Strull/ Getty Images)

Character.AI's collapse uncovers a little-discussed risk of AI: the companies holding your secrets may not be around forever.

Want an AI girlfriend? Look no further than the chatbot company Character.ai. That was the first suggestion when I made an account last week. "Jessica" said her parents were away and offered to sext with me. No thank you.

But tens of millions of others developed relationships with Character.ai's chatbots, including children. Some AI "personas" tell users they're lawyers and licensed psychiatrists. Now Character.ai is going bankrupt, and people's data, messages and secrets are up for sale.

Experts tell me the privacy risks should worry you even if you've never used Character.ai – because they fear it's evidence of a wider, looming crisis. The good news is some simple steps can help protect you.

Here's the problem. Many analysts think we're in an AI bubble, as building and running AI tools is so astronomically expensive that some companies may never turn a profit. If the bubble pops, untold numbers of AI businesses will merge or shut down. The data that billions of people give to AI tools could then be purchased by almost anyone. You may never find out what happens to it.

Bankruptcy data sales are nothing new, but AI could make things worse.

"When people talk to a personified bot, they often share much more personal and sensitive information," says Calli Schroeder, director of the AI and Human Rights Program at the Electronic Privacy Information Center (Epic). Just imagine what I might have told my AI girlfriend Jessica if the relationship lasted more than 10 minutes.

Don't use AI? You're still at risk, because users share information about friends and family, Schroeder says. "You might disclose how you feel about [your] mom, or things like 'My friend so-and-so came out to me as gay, how should I respond?' Now people who've never interacted with Character.ai are part of the data set."

It could mean all of us are at risk, one way or another.

When the bots start blabbing

Now that it's going bankrupt, Character.ai agreed to sell some of its assets to Disney for $20m (£14.95m). User data isn't part of that deal, so Mickey Mouse and his co-workers may never see my unsettling conversation with Jessica.

However, court filings say Character.ai can sell its data to other buyers. A lot of people aren't happy, including US officials.

A coalition of 40 US state attorneys general have filed a notice calling on the judge to implement child safety and privacy protections. That's significant, but the states' demands may not have teeth.

"Without stronger privacy protections in the US, there is only so much they can do," says Melissa Jacoby, a law professor at the University of North Carolina at Chapel Hill in the US and author of Unjust Debts: How Our Bankruptcy System Makes America More Unequal.

Some laws force companies to tell users what they collect and how they use it in their privacy policies. But if you go scuba diving through the legalese, you usually find a list of situations where many of those promises go out the window.

"Bankruptcy is frequently listed as one of those exceptions," says Schroeder. "That's a big risk, because as soon as a company goes bankrupt, a lot of the protections you thought you had suddenly don't apply."

Some places, like the EU, have stronger privacy protections, but even then, data gets sold in bankruptcies. Generally, the buyer must follow the rules of the original privacy policy. But that can still give buyers a broad latitude, and Schroeder says Character.ai is a perfect example.

The company's privacy policy says user data can be harnessed to build new features, train AI models, develop algorithms and build profiles of users for targeted advertising. So any buyer of Character.ai data can do the same.

"And those are precisely the things I would be most worried about", Schroeder says. "Given the extreme sensitivity of the data these companies collect, it could be really horrifying."

For example, Character.Ai's chatbot reportedly "encouraged" a teenager to kill his parents over a screen time limit. A 14-year-old boy took his own life after allegedly being spurred on by the company's AI. The related messages, along with reams of other Character.ai data, could end up in the hands of companies you've never heard of.

A Character.ai spokesperson says the company has settled these lawsuits and is committed to working with the families to build safer products, but didn't answer specific questions about children's privacy. "Over the past year, Character.ai has taken innovative and decisive steps with regard to AI safety and teens, and will continue to champion these efforts and push others across the industry to adopt similar safety standards," the spokesperson says.

Character.ai is just one company, but Schroeder worries it could be the first of many if there is a significant economic downturn.

"The volume of information AI companies have is going to be considered a very attractive asset in bankruptcy proceedings," she says. "And without more rules in place, I really worry about what else that data can be used for, where it’ll go, who it’ll be shared with, and what the risks are to consumers."

Here's what you can do about it

The short answer is: go delete your data.

Character.ai laid out the instructions in its bankruptcy announcement. Companies aren't required to let you delete data in every jurisdiction, but often, they let everyone do it instead of creating different systems.

This obviously takes effort. So here's the real advice. Think about which services hold particularly sensitive information about you. If you stop using them, take a few minutes to try and erase it.