Breaking the news: Fake news in the age of AI
An investigation into AI anchors, AI stations, and the irresponsible false stories they spread
Diving in: Just how big of a problem is AI news?
This investigation started with a text from my dad, a link to a YouTube video depicting an outlandish breaking news story, and a question: Is this AI?? In fact, I received several messages along these lines from friends and relatives, and it got me wondering: How widespread is AI news on social media?
I started to poke around, and I was shocked both by the quality (high and low) and quantity (hundreds of accounts, thousands of videos, tens-of-millions of views) of the content I found over several short investigations. I’ve been doing this sort of OSINT (open source intelligence) work for years, and after talking to my parents and others about coming across this content in the wild, I think it’s important we apply these investigative methods to a rapidly growing problem.
MSNBBC? The frustration of AI news channels
Most AI-generated news segments don’t appear in a vacuum. AI-generated news channels post these segments at an incredible clip, optimizing for engagement through splashy false headlines delivered in deadpan by AI-rendered imitations of famous news personalities. These AI channels have branding, consistent video upload schedules, AI-generated video titles/descriptions, comment interaction, and other consistent on-screen artifacts. In short, we’re seeing mini TV stations spring in and out of existence as they flaunt moderation through sheer numbers and copypasta disclaimers which bury the lede that the personalities are AI and the stories are fake news.
Here is a snapshot of some of the channels and videos that we tracked with aggregate engagement numbers represented in the table below. The graphs throughout this investigation represent our searches for content (purple nodes/edges), pieces of content (blue nodes/edges), and accounts (green nodes/edges) related to AI-generated fake news on major social media platforms.
Across 142 unique AI news stations, we tallied over 700K subscribers, and tens of millions of views, with some videos garnering over 1M.
This investigation exposes a small subset of the AI-generated fake news videos easily found on YouTube and TikTok. I hope it highlights some current harms and helps raise awareness of potential future risks by emphasizing the inflammatory, irresponsible, and often outright false narratives peddled by these accounts.
Many of the videos that started this investigation have been taken down by the respective platform, just to have hundreds more surge in their place. The videos tend to be removed only after they get heavily flagged by users, and many of the videos appear to avoid both moderator actions and synthetic content tags long enough to become somewhat viral.
AI Maddow — No single source of truth
To retrace some of the steps we took, start by quote searching “Rachel Maddow” on YouTube and filter by channel. For me, this surfaced dozens of channels using AI imitations of Rachel Maddow to spread fake news.
Of course, none of these channels are in any way affiliated with American TV host Rachel Maddow. You might note the low subscriber counts and shrug, but this is just the tip of the iceberg. Replicate the search with “[your favorite prime time news personality]”, filter by channel, and you may get something similar.
Most of the videos on these channels have synthetic content labels. But, they are easy to miss (they’re small and disappear after several seconds), and some videos get many thousands of views before they get taken down, including this video which had 66K views featuring AI Maddow spreading a false narrative about Republicans in congress turning on Trump. It has since been removed.
This video by The Maddow Talks (@themaddowtalks) on YouTube had over 417K views and no synthetic content filter when it was taken down.
These AI news segments lag well behind the current state-of-the-art for generative AI video, and that's actually cause for concern. They will get better as the technology continues to diffuse. Unfortunately, a quick glance at comments suggests that many commenters can't tell these videos are AI despite the relatively low quality. Of course, many comments could be fake just like the videos themselves, but if that's true then the problem is even bigger, with an entire ecosystem of fake users boosting this content so that real users are more likely to fall for it.
The “Senate in CHAOS…” video (and many others that I will highlight below) claims that the Senate just impeached Trump. Of course that's false, and the arguments are incoherent. AI Maddow does not bother with the procedural subtleties between filing articles of impeachment and voting to impeach, something fans of the media cycle around Trump's first term will surely be sensitive to. But these videos featuring outright false claims about high-stakes political events are doing everything in their power to convince users that they are real. Some viewers might believe what they see, or at least let the message raise their blood pressure just a bit more.
It should be said that this spat of AI news coverage isn't a one-off messaging blitz around an imagined Trump impeachment. Please enjoy this video where an AI Maddow casually tells viewers that China just invaded Taiwan. Thankfully, it got almost zero engagement; unfortunately, at time of publishing it was still live.
Let’s do a mini-investigation
Riddance has a suite of OSINT tools to better report on and track fake or AI-generated content. As investigations like this one exploded in scope, we built tools to log all of the videos and accounts. Here is a bigger sample of the channels and videos surfaced depicting AI Maddow, every green node is an AI impersonator Maddow channel, with outer blue nodes as sample videos uploaded by the connecting channel node. The purple node at the center of the graph represents our quote search for “Rachel Maddow” channels on YouTube. The table below the graph depicts the results of the search in more detail.
To better understand the breadth and depth of of the fake AI news problem, here are two text strings we used to surface AI news content on YouTube and TikTok around this false impeachment narrative. A string here just means a specific sequence of characters, including spaces. When searching for a string in quotes, the platform will prioritize exact matches.
Search #1: “Congress Votes REMOVAL _ Trump Refuses to Leave!”
I wanted to get a sense of how many other videos and channels were using the exact same video title, and in this case there were at least 20 others. But despite matching titles, they were largely from different accounts, featuring different AI anchors (or different versions of the same anchor) reading largely similar (but not identical) scripts. These videos amassed hundreds of thousands of views.
For example, this video, which featured an AI generated George Will (in general the fake AI news phenomenon is not a partisan problem) from the YouTube channel @GeorgeNewsAnalysis had over 172K views before removal.
Replicating the same search on TikTok ("Congress Votes REMOVAL _ Trump Refuses to Leave!") yielded similar results. A collection of different accounts posting different videos all with the same title and a nearly identical script.
Despite many posts with low view counts, all it takes is one video to go viral for the effort to pay off. People making these bulk AI news videos price that into their approach. It means the standard mechanisms for moderation are not very effective here.
Here's a similarly themed video, from another YouTube channel (@CivicStorm-1) with over 300K views and no synthetic content filter.
If we take a look at the network of accounts posting videos with this title, we see that there are even more views, and even more channels, all circulating the same false narrative that the senate just successfully impeached Trump.
Search #2: “IT’S OVER: Senate Turns on Trump in Stunning Midnight Vote - White House Goes Dark!”
On YouTube this search returned 38 distinct channels and 42 unique videos, with different AI personas delivering the news. Some of these personas are copycats of primetime anchors, others are completely fabricated anchors or bloggers delivering this completely fabricated news to an audience that is at least partially credulous. The table below shows results for videos in this network, by view count.
Advice for Readers — Before assessing truth, ask yourself: is this real?
The investigation above was just a very brief look into the messy and viral network of fake AI news content. Its clear that platforms are working hard to moderate this activity, but the ease with which new channels crop up combined with their rapid posting cadence makes it feel like a losing battle. More robust solutions are necessary and hopefully techniques for surfacing these networks can contribute to solving the problems posed by fake AI news.
At Riddance, we want to give people tools to reliably converge on true beliefs. Sometimes, answering the question ‘is this true?’ is harder than answering the question ‘is this fake?’ Pulling those questions apart can help hack through uncertainty, especially online. Learning that an information source is fake is often grounds to disregard the information from that source, particularly when the information is intended to be public. If you want to know what news to trust, a good first rule might be the following: Don’t trust any AI-generated news content, personas, outlets, influencers, or reaction videos. It’s fake news.