A network of inauthentic TikTok accounts created deepfakes of Tucker Carlson that spread across TikTok and X, poisoning Google’s AI Overview and search results. By May 5th, they had accumulated over 38 million views with videos using Carlson’s likeness and collected over 200,000 followers.

Riddance provided TikTok with a list of accounts, which have since been removed. They were typical for the types of inauthentic accounts that every major video platform deals with.

People send me tons of AI videos through direct messages and emails, but I found these in the wild. A couple of days after the White House Correspondents’ Dinner assassination attempt, I saw a deepfake using Tucker Carlson’s likeness, hinting at a conspiracy and failure of leadership.

I wanted to compare the AI-generated script to what Carlson had actually said. So, I Googled “tucker carlson white house correspondents dinner”.

Google’s AI Overview said that “Tucker Carlson and others spread, via online video, ‘staged’ conspiracy theories regarding the event.” The first Search result, from The News International, reported on “A video of Tucker Carlson floating the ‘staged’ conspiracy theory.” This story was the main source in Google’s AI Overview.

The News International’s story was wrong. It sourced a deepfake that looked extremely similar to the one I found. This one was posted on X, received almost 280,000 views, and looked incredibly similar to the deepfake I had seen. This story lazily reported on the spread of this video, unsure whether it was real.

Subscribe now

To date, there’s no record that Carlson has made a public statement about the White House Correspondents’ Dinner assasssination attempt. Google’s AI interview was indirectly poisoned.

This particular campaign focused on posting to TikTok, while reposts to other platforms appeared to be unaffiliated. A brief investigation uncovered eight TikTok accounts in the network, coordinating posts with the same scripts or the exact same videos.

These accounts posed as Tucker Carlson fan pages or clippers. They often reported real news and mimicked Carlson’s actual views and talking points. But they also included occasional fake news, like fabricated stories about US ships being destroyed in the Strait of Hormuz, or content that exaggerated China’s or Iran’s narratives of the Iran war. Their specific claims were often difficult to search and validate, and that was the point.

Their Carlson deepfakes were relatively high quality. They accurately mimicked Carlson’s unique cadence, aside from the occasional phrasing error (saying “twelve dollars million” instead of “twelve million dollars”). These videos used different source clips from Carlson’s show, alternating his clothing to avoid repetition. They included video cutaways of news events or real footage, too. This shows that real editors had a hand in some of these videos.

Referencing a previous article, TikTok noted that they “work diligently to get ahead of these accounts, and proactively prevented over 350 million fake accounts from signing up in 2025.”

This network used Tucker Carlson’s likeness to frame real news and fake news with a bipartisan, anti-establishment voice. They were convincing enough to confuse humans, AIs, and search algorithms alike.

A large collection of stop motion and animation artists had their channels demonetized or banned after YouTube flagged their content as inauthentic. These accounts, which do not use AI, appear to be caught up in a wave of enforcement as YouTube tries to restrict inauthentic content.

Creators have struggled appealing these decisions, and many have been unable to get the penalty reversed. Some have needed to turn to public statements to get YouTube’s attention.

Peter Heacock and Marie Hart are stop motion animators who run Tiny Grandma, where they use a puppet inspired by Marie’s mother to teach viewers how to make Korean cuisine. YouTube demonetized their channel citing “content that appears mass-produced or repetitive, without adding significant educational or other value”.

Peter and Marie were baffled. In a response video, Peter said “I don’t think you can get any more hands-on or low-tech than frame-by-frame moving a puppet and having it come to life.” They submitted an appeal right away, and YouTube reversed their decision a few days later.

But that speedy resolution isn’t a given. Chris Plim runs Plim Stopmotion, where he posts stop motion tutorials. After multiple failed appeals through YouTube’s system, he resorted to making his own response video. The channel was finally remonetized shortly after.

Corey Nikolaus runs EraserDread, which was entirely removed by YouTube in January, with the platform citing their inauthentic content policy. YouTube didn’t stop there, removing two other channels linked to his account. One of the accounts called “Monster Circus” was removed before he could upload content to it. Corey has effectively been blocked from using YouTube as a creator. He has submitted multiple appeals, with no success.

These aren’t isolated incidents. The Mr Skelly account has also been demonetized, despite behind-the-scenes videos proving its authentic content. Same with The Good Times Are Killing Me, another popular digital animation page. There are multiple Reddit threads of real creators struggling to get their accounts back, all reporting being labeled as inauthentic accounts.

As AI videos copy handmade and digital art styles at scale, audiences are pushing back and expecting platforms like YouTube to moderate AI content. But their methods are backfiring, harming the creators they should protect.

These broad-brush attempts to protect audiences from deceptive material accelerate the encroachment of AI-generated media. Animation and stop motion communities, already competing with AI media for viewers, are now fighting faulty moderation efforts as well.

YouTube did not respond to a request for comment. But as they moderate content at scale, combing through an average of 20 million videos per day, it’s safe to assume that AI moderation is used to keep up with the volume.

But these AI systems are not sufficiently context-aware to make critical judgements about authenticity. These systems need human oversight and intervention, especially when livelihoods are at stake.

If our small team at Riddance can conduct brief investigations to validate if creators are real, should we not expect YouTube to have a similar capacity? As a profitable organization who releases AI video generators right into their platform, they need to learn from this mistake and allocate more resources towards authenticity review.

In the meantime, consider following these artists here:

TikTok:

• @itsmrskelly

• @tiny_grandmom

• @plim.stopmotion

• @good_times_show

Instagram

• @chris_plim

• @good_times_show

• @tinygrandmom

• @1924us [Mr Skelly]

• @eraserdread

I doubt that you’ve heard of KUGIG, but you might have seen their AI ads without realizing it.

An oxygenated, comedically-bandaged woman lays in a hospital bed. Her husband cries over her, kisses her cheek, and whimpers, “I’m right here baby, I love you.” The woman could die… if she hadn’t followed her doctor’s advice and switched to KUGIG toothpaste.

“Wanda Dear898” commented, “I switched to KUGIG toothpaste before, and to be honest, my gums are no longer so sensitive now.” This comment has a long thread of responses, each talking about how KUGIG toothpaste fixed their swollen face, yellowed teeth, or toothaches.

This absurd, spammy AI video had over 3.1 million views before it was removed. It’s one of many AI-generated ads made for KUGIG. These videos are dumb, sloppy, and manipulative, but there are so many of them that they’re probably working.

This is an unregulated and unlabeled advertising campaign making unsubstantiated and misleading claims about healthcare products. But even in a sea of spammy AI ads, KUGIG’s stand out.

The Riddance team watched over 200 of them, tallying over 11 million views. These videos came from 57 accounts, but there are certainly more accounts and videos out there. We had to stop somewhere, so we only watched and tallied a few videos per page.

Riddance reached out to TikTok for a comment and provided them a list of confirmed KUGIG-affiliated accounts. Within three days, all of these accounts were offline. But despite our reporting and TikTok’s response, there are many more KUGIG-affiliated accounts and videos popping up every day.

It’s not “organic” for swarms of accounts to only make AI videos about a small hygiene brand sold exclusively through an Amazon storefront called “HKKANGXING” and a few TikTok shops. Their Amazon storefront is based in Hong Kong, and their TikTok shops are based in Henan, China.

How does this happen, and how do large campaigns like this fly under the radar for so long?

Subscribe now

How the videos are made

The “woman in a hospital” video type is just one of the of many bombastic, AI-generated videos in the campaign. Other examples include:

1. A woman sobs in a dentist chair. She has a huge bulge on the inside of her cheek. Her teeth are black and rotting as a dentist tool pries at her cheek. The caption includes “my dentist told me to switch to KUGIG Toothpaste.”

2. A Costco-esque store is overrun by crowds fighting over KUGIG wash, piled high in a pyramid. The caption is “My God! Please stop fighting over it!”

3. A young woman cries and complains that her “whole face is on fire.” A woman (presumably her mother) opens her mouth to reveal disgusting teeth. The caption reads, “She’s complaining that I didn’t pick out a good toothpaste for her.” The comments include a planted comment recommending KUGIG toothpaste.

There are also AI-generated selfie testimonials from teenagers, “Pixar-styled” AI advertisements, fake doctor testimonials, and more.

Two accounts clearly demonstrate a unique trait of the KUGIG campaign. Both have the bio “We listened to the doctor and switched to KUGIG Wash. No more issues now.” Rather than posting exactly identical AI videos, they post nearly identical AI videos. For example, they each posted two videos of a black man with dreadlocks. These four videos were probably all from the same batch-generation output of an AI video generator.

A prompt like “A black man with dreadlocks, a white t-shirt, and grey Nike sweatpants scratches his groin in the bathroom” could yield all four of these videos at once, because AI video generation is a bit random. This method was used extensively across the campaign’s many video types because it’s resourceful and helps them avoid spam detection.

How the accounts are linked

What accounts lack in duplicated videos they make up for with duplicated titles and hashtags. The caption “Dental Care Expert! KUGIG TOOTHPASTE” is both awkward and common. Remember, this isn’t an exhaustive sample of videos, so there are many more videos with that caption.

It’s sensible that videos in coordinated campaigns will coordinate hashtags, too. We saw nearly twice as many videos pushing KUGIG toothpaste as KUGIG wash, even though many accounts pushed both products. The hashtags reflected that balance.

Many of the accounts were hilariously strict with comment moderation. One video had over 1.1 million views, but only 40 visible comments that all promoted KUGIG, presumably from coordinated bot accounts.

It’s very common to see fake accounts change hands between campaigns. Unlike human influencers, accounts with AI-generated content can make these pivots quickly and shamelessly.

One account was spamming AI slop of Buddha statues and Hello Kitty Cruises before it pivoted to AI videos of women wrapped up in blankets while crying in pain. Another account participated in an unrelated campaign for a healthcare supplement before pivoting to KUGIG products. Another went the opposite direction by starting with KUGIG videos, then switching to supplements during our research period.

What can be done?

These accounts break many of TikTok’s rules all at once, across their Community Guidelines, Advertising Policies, and Terms of Service. But moderating them is a daunting challenge.

While some of the videos and accounts were removed during our investigation, many of the most popular videos stayed up. The KUGIG-affiliated accounts don’t self-label AI videos. And yet, seven of the top ten videos by views were labeled with the “Contains AI-generated media” tag. This means that TikTok’s automated systems or manual reviews caught that these videos were AI-generated.

But this expansive campaign ventures into the “influence operation” space more than most spammy ad campaigns we’ve seen. It shares similarities with more sophisticated campaigns that make it harder to detect.

In a response to our report, TikTok qualified these accounts as “spammers [that] post AI or low-quality content to chase engagement, which is unfortunately a common issue across every social media platform.” They stated that they “work diligently to get ahead of these accounts, and proactively prevented over 350 million fake accounts from signing up in 2025.”

KUGIG resembles the drop shippers who use AI videos to pretend their products are handmade. It’s that playbook, scaled up for a “healthcare dropshipper,” using dozens of accounts to amplify manipulative AI videos. The AI-enhanced future of spammy, manipulative advertising is already here.

At Riddance, we’ve observed a pattern in our investigations: When two or more AI systems overlap, the guardrails designed to reduce harm and prevent Terms of Service (ToS) violations stop working like they should. This failure is not an edge case but rather points to something structural about the difference between human labor and AI labor when accountability matters. Now that AI is increasingly capable of replacing tasks that once required human minds to carry out, we must scrutinize the extent to which the accountability structures that inform human actions are applicable to AIs acting in those same capacities.

A double-sided failure

In our investigation into AI-generated sexualized videos of minors on Instagram, we identified content that, in our opinion, clearly violates Meta’s policies on child sexual exploitation. But given how the videos were made—stitching together AI scenes that were not necessarily problematic in isolation—the videos could have been generated by commercial models without hitting guardrails. And, because platforms automate a lot of moderation, the pedophilic innuendos in these videos went unnoticed. Human beings could have caught this behavior, but the existing systems failed.

AI-generated news accounts that spread false or misleading information about the most pressing current events present another opportunity for accountability failure. These channels impersonate real newscasters but inherit none of the consequences when the stories are false or drastically overstated. While platform-specific moderation flags much of this content for demonetization, if the actor has a political motivation rather than a financial one, these actions will barely deter them. The AI models making the videos are not jailbroken, and technically none of the content is in violation of platform-side ToS. From the perspective of the AI either generating or moderating the content, there is no problem, yet clearly this content is problematic, as evidenced by platforms trying to manually remove it.

This double-sided failure is a microcosm of the accountability gap writ large. The AI that made the content had no mechanisms in place to understand the harms it perpetuated, and the AI moderating the content had no general concept of what those harms look like in the first place. AI is not sufficiently sensitive to reasons for the actions it does or does not take, whether the domain is maintaining critical infrastructure or removing CSAM-adjacent content from short-form video algorithms. Right now, we are hyper-focused on outcomes in AI systems and not paying enough attention to reasons that guide actions taken to effect outcomes. In order to have safe AI we desperately need both.

Subscribe now

Accountability requires sensitivity to reasons

Human beings constantly navigate an intricate space of reasons. Some examples include: choosing not to invite a friend to dinner because you know they do not get along with another invited guest; making a plan to help break a bad habit; changing behavior because someone asked you to do something differently. Two distinctive qualities of the reasons guiding human action are how broad they can be and how connected they are to the effects those actions have in the world. The narrowness of the reasons guiding AI actions coupled with their inability to perceive downstream consequences are just two of the key differences between reasons guiding human action and the reasons guiding AI action.

Reasons for actions and accountability are intimately linked. The reasons that guide our actions are often backstopped by accountability, and accountability itself always requires sensitivity to reasons. Social rules, laws, and guidelines are norms that we adhere to if we agree with the reasons behind them. Of course, sometimes rule-following happens because we fear the consequences of breaking a rule—if I drive too fast I’ll get a ticket—but the best rules are generally those that we follow because we assent to reasons supporting them, not to an auxiliary reason riding alongside them. Whether we adhere to norms because we are self-interest maximizers (consequence fearing) or because we think some norms are themselves worthy of adherence, we have a sense of accountability as we make a decision about following them.

When training an AI model, all sorts of rewards and feedback get used in an effort to minimize a loss function or optimize for some behavior. In addition to training for performance, this is an attempt to bake in awareness of specific guardrails or norms. But once a model is deployed, that feedback mechanism vanishes. The real, messy world of interconnected reasons and norms has no purchase on the decision-making calculus of a model. Furthermore, beyond their context window, AI models are completely blind to downstream effects of their actions.

Over the course of a session, one can tweak the prompt context to adjust for more or less desirable outcomes, but there is no reflective moment where an AI can pause and assess the stakes or meaningfully feel responsible. It’s not even clear what such a reflective moment would look like for an AI, and as a result AI models lack accountability in a very structural way.

This lack of accountability might seem like a good thing. After all, AI is not human, and deflecting responsibility for actions to non-human systems might look like cop-out at best and a deliberate tactic for avoiding consequences at worst. But if we are willing to offload increasing amounts of human labor—much of it increasingly impactful—to AI, then we have to seriously confront the accountability gap that follows.

Last week, OpenAI announced that they will be shutting down the Sora app on April 26th. This makes sense, because Sora was a bad idea.

Sora was still innovative. The Sora 2 video model was funny and made social media-style videos, rather than boring “filmic” AI videos of the time. The Cameo feature made deepfakes fun, and though I’m still not over how damaging Cameo was, it gave people FOMO and built hype around the Sora app.

AI videos are expensive, and OpenAI paid for a lot of them. Since Sora was like TikTok if it only had AI videos, the app didn’t appeal to normal people. So, Sora’s users were incentivized to take those videos and post them literally everywhere else. Sora subsidized slop for everyone.

I don’t think that’s the main reason OpenAI is shutting down Sora, though. The Sora 2 video model wasn’t competitive anymore, and OpenAI would have to invest into keeping up. Instead, they’re focusing their attention on world models and large language models, which they should have never been distracted from in the first place.

Subscribe now

The Sora models are outdated… already

OpenAI will also shut down Sora’s API on September 24th. Through the API, developers and creators can pay to generate AI videos with the Sora 2 models.

The baseline Sora 2 video, which is the default video inside the Sora app, is a 10-second, 720p video that costs $1.00 to create. In other words, any time you make a “free” video inside the Sora app, OpenAI “pays” $1.00 of API resources to create it for you. If you would rather get a video without a watermark, you can pay $1.00 through the API (or use a website like fal.ai or Higgsfield to do it for you).

Moreover, Sora 2 videos don’t look great by today’s standards. Mind you, Sora 2 was released barely six months ago. It’s fine for casual AI slop, but even creators who make “good” AI slop want something better now. But, the improved Sora 2 Pro model costs $3.00 for a 10-second video at 1080p quality.

Another mainstream AI video model, Google’s Veo 3.1, is more expensive than Sora. In my opinion, Veo is even worse. But Kling’s o3 video model is better quality than either Sora or Veo, and it’s cheaper.

Below is a benchmark I run with video models that I call “Four people talking.” Each character is supposed to introduce themselves, from left to right.1 It tests prompt adherence. Sora mixes up lines, and Veo mixes up lines while adding in an extra person. Kling o3 is the first model to ever pass this test.

The most popular types of AI videos, and those that are most profitable for the creators and grifters who make them, are rarely made with Sora 2 anymore. Alibaba’s Wan video model makes sexual content. I’m not sure which video model made AI Fruit Love Island, but it certainly was not Sora 2.

Sora’s models specialized in social media videos like fake Ring doorbell cameras, bizarre scenarios, and engagement slop. Kling’s o3 model can do these better and cheaper, so I’ve seen a lot of slop pages make the switch. And with pressure from the upcoming Seedance 2.0 — which is still mysteriously delayed in the US and Japan — Open AI needed Sora 3 to come very quickly.

To be clear, the talented Sora team might have made Sora 3 the best of them all. But at what cost? Instead, the team will refocus on virtual world models. If you read the original Sora technical report from February of 2024, that was their original goal for Sora all along.

The AI video race doesn’t matter

Sora’s closure is not a signal that the AI bubble is about to pop. It’s a signal that AI video generation is a shaky business model without the upside of large language models. During Sora’s 208-day lifespan, OpenAI lost a lot of ground to Anthropic where it matters. Anthropic never got distracted by AI video.

There’s little doubt in my mind that “AI” is bad for the world, and that there’s an “AI bubble.” I’m not optimistic about an AI-dominated future. But, I often see these legitimate criticisms bundled with the assertion that “AI” is still bad, or that only stupid people use AI.

This isn’t a legitimate criticism anymore. Large language models have improved a lot in the past six months. I used to be very skeptical that AI would ever be useful. But, after seeing Claude’s Opus 4.5, released in November of 2025, and its current iteration at Opus 4.6, I’m confident that LLMs will be economically useful for a long time.

When OpenAI released Sora in October, LLM progress felt stagnant. Since then, Gemini 3’s release triggered a “Code Red” memo at OpenAI, and now Anthropic is probably winning at code generation overall.

GPUs can be used to train video models, or large language models. The highly-publicized $1 billion deal between OpenAI and Disney will fall through, but that money is nothing compared to the money OpenAI needs for large language models.

The bubble might still pop. OpenAI’s financials are still disconnected from reality. But there is industry-wide momentum in coding and enterprise solutions, and OpenAI can’t waste GPUs and resources on Sora any longer.

On the afternoon of February 2nd, I received an email from a student researcher named Angel Nulani. She had found 40 social media accounts with AI-generated black women that perpetuated racist, anti-black behavior. Coincidentally, earlier that day, I had received an email from Sharihan Al-Akhras from BBC News Arabic. She was working on a piece about an impossibly-black AI-generated character that stole real videos and “reskinned” them with AI.

Together, Riddance embarked on an investigation in collaboration with the BBC that uncovered over 100 social media accounts that depict the likenesses of Black women through AI-generated characters.

Research revealed accounts running out of 34 countries across North and South America, Europe, Africa, and Asia. They tally a cumulative 19.4 million followers across Instagram and TikTok. The majority (68 accounts) actively promote monetization on sites where they sell explicit AI-generated media.

AI-generated videos are especially adept in race-to-the-bottom trends because creators can avoid accountability. They push limits to get attention. But by depicting black women, they perpetuate a long history of exploiting black people, using caricatures and self-hating characters to attract morbidly curious or racist audiences.

By identifying and examining these accounts, we can start to find solutions for this quickly-growing problem. With AI media improving rapidly, solutions must keep up, otherwise this AI-perpetuated exploitation will only get worse.

How AI advancements scaled digital blackface

Our sample of 100 accounts is representative of both big and small accounts. Of the 100 accounts, 63 were exclusive to Instagram, 4 were exclusive to TikTok, and 33 had pages on both. This means that tracked 136 pages total.

Most of the Instagram accounts were created between mid-2025 and early 2026. A third were created in the past three months alone, and 60 changed their usernames at least once, meaning they “pivoted” to AI-generated black women. The trend is clearly demonstrated in the graph below, measuring when the Instagram accounts were created and if they still have their original usernames.

Google released the Veo 3 AI video model in May, and companies like Kling and Runway unveiled competing video models shortly after. AI-generated videos flooded Instagram and TikTok at this time, as creators experimented with AI slop but lacked clear monetization plans. Once their accounts grew, they looked towards profitable AI niches.

All 100 accounts had to satisfy three criteria:

• They use entirely AI-generated media

• They depict black women (including numerous albino and vitiligo accounts)

• They are sexual in nature

The explosion of accounts from late 2025 through today directly correlates with the releases of open-source AI generation models: FLUX-2, an AI image model launched in November, and Alibaba’s Wan 2.2 Animate Replace, an AI video model released in September. FLUX-2 can generate sexual images, and Wan 2.2 can “animate” them. This combination lets AI creators generate realistic, consistent AI videos without guardrails.

Subscribe now

How the accounts depict black women

Many of the AI-generated characters demonstrate explicit, internalized anti-blackness. They may state a desire to be European (particularly Slavic) or ask if they’re “good enough” for white attention. Women with vitiligo and albinism are characterized as having one white parent, outright calling themselves white or half-white. At least three accounts have made references to the n-word, and one gives its followers an “n-word pass.”

Women are often depicted kissing famous white men like Donald Trump, John Cena, or Johnny Sins (a white male porn actor). Some accounts post about Jeffrey Epstein, with one character attempting to defend him. (35) of the accounts include race-play terminology in their usernames, like “black”, “white”, “vanilla”, “charcoal”, “dark”, “ebony”, or “noir”.

Many accounts fetishize aspects of Black culture and the Black diaspora while having no understanding of them. Black American naming conventions are used for African characters. African accounts often have no specified ethnicity. And when they try to give characters a cultural background, they get basic information wrong. For example, Carolamater, an account based in Lithuania, depicts a Mursi woman with light skin and box braids. The real Mursi people, from southern Ethiopia, are known for their distinctive lip plates and body painting.

And then, there are the characters who have impossibly-black skin.

Not normal

What does it mean to have “impossibly-black” skin?

Part of my job is to perfectly dial-in accurate skin tones on camera. This helps me identify exactly what’s wrong with these characters. Any real human, light or dark-skinned, has skin tones falling on a vector of various red-orange hues. This is clearly demonstrated by a vectorscope, which analyzes color and saturation.

Below, I’ve highlighted four dots representing different skin tones. These all cluster on the top-left vector around what is known as the “skin tone line”, which slopes up and to the left.

Impossibly-black characters, on the other hand, land almost directly in the middle of the vectorscope, which means there is no color saturation in their skin. Notably, this is a separate measurement from the amount of light reflected off the skin, which can also vary. The resulting characters display a range of dark grey to pitch-black skin.

Historically racist portrayals of Black people, like caricatures or minstrel shows, often painted their faces literally black. And if it’s not obvious, AI-generated characters will have normal skin tones by default. Human creators have to deliberately try to make impossibly-black characters. In testing, we have confirmed that prompting FLUX-2 for “very dark, pitch black skin with no undertones” will yield impossibly-black skin.

Nianoir is a character with impossibly-black skin. It had the most followers of all 100 accounts, with 3.1 million on TikTok alone — 2.7 million more than the next largest TikTok account. The account was removed from TikTok after our request for comment, but remains up on Instagram (we’ll dive into the platform responses later).

Their bio says “Just a girl with dark side... 🖤” [Grammar incorrect]. The Ukraine-based account makes many of their videos by stealing content from real creators, then replacing them with the Nianoir character. The AI video model Wan 2.2 Animate Replace does this by design.

The BBC interviewed Riya Ulan, a young model from Malaysia, who had her own videos stolen by Nianoir. Ms. Ulan was stunned and tried to take action, but was unable to get the stolen content taken down until the BBC reached out.

The sad irony is that Nianoir’s comment sections are full of people posting screenshots from the videos Nianoir stole from — often from white or Asian creators like Ms. Ulan — which are met with accusations that these creators are the ones whitewashing.

Organized, international operations

AI-generated media is about quantity before quality, so operators often run multiple accounts. Sometimes, this turns into full operations with teams of people working in content farms.

One operation, based in the United Kingdom, runs ten accounts, including four of the top ten in total followers. Their accounts totaled almost 3.5 million followers, though there’s significant follower overlap. These accounts follow one another, “collaborate” between characters, and some use the exact same character.

Their accounts include zurilovesvanilla, the fourth-most-followed account of all 100, and ayannasoblack, an impossibly-black character. Naledi_the_white_girl is an albino black character who once posted that she “still commits 50% of the crime.”

Many accounts had at least one former username, showing this group was making other AI content before pivoting to Black characters. Eighteen username changes before landing on “zurilovesvanilla” tells you this wasn’t their first idea, but it was their most profitable one.

How do they make money? The majority of accounts (68 of the 100) promote monetization links to sites where they sell AI porn. You can assume that the other 32 are waiting to get big enough to monetize. None of the content on social media is explicit, so the objective is to capture an audience on an social media, then filter the “whales” — a term from betting meaning high-spending individuals — to the monetized sites.

A group based in Kosovo demonstrates the recency of the trend. They run seven accounts and focus on dark-skinned, curvy avatars, with a couple (shaykelt and nyxkash) having the same exact avatar. Their accounts combined to a total of 680,000 followers. Every account was created in the past two months and has no username changes. This group identified the niche and went all-in right away.

What platforms can do

I wasn’t expecting Instagram or TikTok to proactively remove many of these accounts, and for the most part, I was right. Over three weeks of collecting and tracking accounts, a few were taken down. But this was insignificant compared to the new accounts that popped up every day.

Then, Riddance and the BBC independently reached out to TikTok and Instagram for comment, and provided both with lists of accounts.

Within three days, TikTok banned 20 of 37 accounts, including removing Nianoir.xo, the single largest account on either platform. Several accounts belonging to the UK group (onlyskyewhite, zurilovesvanilla, emma.cynder) and the Kosovo group (nyxkash, abbyblacki9) were taken offline entirely. TikTok also applied AI-generated labels to the videos we shared. These were significant steps, taken quickly.

Instagram took down just 2 accounts: ayannasoblack and naledi_the_white_girl, both belonging to the UK content farm. Nineteen other accounts were age-gated, meaning they went from fully public to requiring digital ID to view. The remaining 59 accounts are still fully public and unchanged.

When asking TikTok which policy those 18 deleted accounts had violated, they cited their policy that prohibits “pretending to be a fake person or organization with the goal of misleading people.”

Instagram’s version of this policy, “Authentic Identity Representation,” prohibits accounts “created or used to deceive others.” But in practice, the policy targets accounts that impersonate real people, or networks engaged in coordinated manipulation. It doesn’t appear to be designed for scenarios like this, when someone creates an entirely fictional persona.

95 of the 100 accounts did not disclose they are AI-generated anywhere on social media. None of them labeled individual posts as AI-generated, which both TikTok and Instagram require. In theory, when either detect a C2PA “watermark” that indicates a video is AI-generated, they can label it as such. But the open-source AI models powering this surge, FLUX-2 and Wan 2.2, don’t embed C2PA metadata.

TikTok stated that over 1.3 billion videos have been labeled as AI-generated to date. They also stated that 98% of the videos TikTok removed for violating edited media and AIGC policies, and 90% that violated sexual activity and services policies, were removed proactively in the most recent reporting period. Instagram declined to comment.

Instagram’s hateful conduct policy explicitly bans harmful stereotypes “historically linked to intimidation or violence, such as Blackface.” The impossibly-black characters documented in this article — characters with no undertones, no color saturation, skin that reads as literal black — are digital Blackface. In my opinion, Instagram’s written policy should cover them.

What’s missing?

Platforms should directly address AI-generated impersonation of marginalized groups. A user who encounters an impossibly-black AI character posting watermelon memes has to decide whether to report it as “hate speech” (which doesn’t capture the AI deception), “spam” (which misses the harm), or “AI-generated content” (which also misses the harm). The reporting categories don’t reflect the problem.

As agentic AI content pipelines become more common, we’ll need automated and human interventions. Automated systems don’t have the depth of social awareness needed to catch what’s happening here. So while platforms will naturally focus on automated, software-based solutions, they should work with third parties with human investigators to help out.

If you come across harmful content, report it for hate speech or sexual exploitation so platforms have that data. It might feel like the reporting goes unnoticed, but in aggregate it may help.

But while AI detection is still possible for some people, the quality improvements of AI media will outpace detection in the long run. The accounts in this investigation are already more convincing than they were six months ago. Platforms must act now by banning accounts and finding algorithmic patterns outside of the videos themselves.

“We’re so beautiful”

Angel Nulani, the student researcher who brought many of these accounts to our attention, did hours of research for this project. As a Black woman herself, this investigation was emotionally taxing. We asked her to be an editor and contributor to the piece. Angel is not a journalist or investigator (yet), but a student with an incredible drive and attention to detail. We assumed she was already a working professional.

As the investigation neared its conclusion, we asked her about how the content and investigation had affected her. We decided to include her response in full.

What upsets me is not that these characters are self-hating, but that there is no “self.” For the majority of the people who are behind these accounts, the only Black people they know are the women they generated. They were not born Black. They chose to be Black, and yet they spend so much time distancing themselves from it.

An ‘all lives matter’ shirt can be paired with booty shorts; crime statistics can be overlaid on thirst traps; videos crying over their features are spliced between suggestive yoga poses.

It’s not enough to pander to people that say “ebony” more than “African” because the appeal is not simply a Black woman desiring white men. The appeal is in a Black woman desiring a complete absorption into whiteness. The men they attract are not suitors, they are saviors.

I’ve tried so hard to disassociate. As a Black woman that’s been online since I was young, I assumed I saw it all. Nothing could really hurt me. At first, all I wanted was to be objective, and rational, and done with it. But this cuts me down to my core.

Ironically, that’s even more confirmation they could never be us, because when you’re real there’s no separating yourself from this. The watermelon emojis are par for the course for them; they’re not for me.

I’m terrified to think there’s a little Black girl somewhere feeling as terrible as I do right now because some guy in Malta wanted €3 a month.

They make us seem so ugly, but we’re beautiful. We’re so beautiful.

A sitting head of state had to post a proof-of-life video this week, responding to online rumors that he was dead and his government was using AI to cover it up. Call me crazy, but I think that’s a really big deal.

This happened because a video address by Israeli Prime Minister Benjamin Netanyahu was opportunistically screenshotted where it looked like his palm was a sixth finger. This wasn’t an AI generation artifact. In fact, I talked about how AI videos mostly fixed hands in my first-ever TikTok video almost a year ago. The AI accusations didn’t come from credible sources, but that didn’t stop the general public from getting really confused.

Pixel-peeping skeptics and attention-seeking creators are finding false-positive “AI artifacts” in every video that Netanyahu has posted since then. Many of the publications that arrived at this conclusion have been accused of being paid off by Israel or AIPAC.

Some commentators are dismissing this as an antisemitic conspiracy. But we can’t dismiss that the public deals with AI videos every day. Their confusion in this case is a natural reaction to their information environment.

This should be a major wake-up call for leaders in tech and government. I see this distrust every day in my inbox. It’s spilling out into the real world, and it’s getting worse.

Figuring out what’s real

Why am I so sure that Netanyahu’s address is real?

It’s a very long, single-camera take (eight minutes and twelve seconds long). That would be an insanely, nearly impossibly long AI video. Even an AI video that’s 20 seconds long struggles with temporal consistency. And without any visible edits or other AI artifacts, it would have to come from a theoretically very advanced AI model — one that doesn’t exist in public yet, and one that certainly wouldn’t add a sixth finger if it existed privately.

Read more

In January 2026, Sweden’s Saab, an aerospace and erstwhile car manufacturer, submitted a proposal to sell 72 Gripen fighter jets to Canada as an alternative to American F-35s. This is a notable development in defense circles, but it’s a relatively niche story. So, it was somewhat surprising to find a coordinated inauthentic behavior (CIB) campaign discussing the Gripen deal almost nonstop and fabricating key details along the way. It turns out these videos were part of a much larger, sustained effort by this network to push content aimed at (further) dividing the US and Canada.

We found 83 videos about the Gripen deal across 46 different accounts, all delivered by AI-generated news anchors impersonating real television hosts. The accounts were usually less than a week old, many of them falsely claiming a deal for 120 jets had already gone through. However, these Gripen videos are a small subset of the 930 videos, from 174 accounts across YouTube and TikTok that we collected during this investigation.

In the graph above, blue nodes represent searches for video titles, green nodes represent accounts, and purple nodes represent videos or posts. The blue edges indicate search results and the green edges represent account ownership of a piece of content.

While Riddance has analyzed a subset of the accounts in this network before, this deeper dive revealed several new operational and narrative dimensions, as well as possible leads on attribution. The videos that we analyzed accumulated over 18M views and 780K likes, with much of the content only days old at time of reporting. YouTube and TikTok appear to be actively moderating this campaign, as several channels were actioned (removed) during the course of our investigation.

Subscribe now

Two things stand out about this network:

• Messaging sophistication: Almost all of the channels combine real breaking news with either exaggerated or else outright false content, causing frustration, confusion, and misplaced anger from viewers and commenters.

• Messaging discipline: In general, the AI news stations we analyzed focused almost exclusively on further distancing the US from its allies with a particular focus on Canada.

The channel below, one of several that take on the entirely-AI generated “Bradley Madden” persona, delivers daily reports about US-Canada relations that are largely fabricated. Other channels position themselves as if they are based in Canada and broadcasting to a Canadian audience.

A similar set of channels on TikTok amplify and modify similar messages. This account, Daily USA News, is representative of the TikTok side of the campaign.

The videos we analyzed often have overlapping titles (and scripts) with other channels in the network, even when the AI anchors delivering the news are totally different. This suggests that the scripts themselves are distributed to the channel makers, who then have a degree of autonomy as to how they produce the videos.

The operation has a least five distinguishing characteristics:

• Fake outlets that siphon credibility from real sources: Channel names like MSN Show, BCN News, News Nation 24, all have the feel of real news stations, and the channel branding follows suit.

• AI anchors impersonating real hosts: This campaign includes impersonations of prominent newscasters as well as fabricated personalities. Impersonated personas include Rachel Maddow, George Will, Tucker Carlson, and Jimmy Kimmel.

• Identical scripts across accounts: As the diagram above demonstrates, there is a high degree of overlap between video titles. We suspect that video script overlap may be even higher than video title overlap.

• Cross-platform distribution: The campaign exists on both YouTube and TikTok, and while these operations don’t look identical across platforms (videos seem to originate on YouTube and then move to TikTok), we identified at least 66 exact content matches between the two platforms.

• Inflated engagement numbers: many accounts seem to have artificial follower numbers, some in the tens of thousands despite being only days old. Of the 39 most prominent YouTube channels in the network, 87% uploaded their first video in 2026.

Like before, most accounts and videos didn’t go viral, but the ones that did generated thousands of comments, tens of thousands of likes, and hundreds of thousands of views. It’s also worth noting that some of the AI is decisively better than in previous campaigns, with commenters rarely identifying the clips as AI generated.

The 20 second clip below is from the video these commenters were reacting to, and it’s substantially higher quality than AI Maddow videos that we covered just a month ago.

As you would expect, higher video quality translates into more engagement, and more accounts in the network will likely follow suit, which means even more difficult detection work for platform-side moderators.

A combo of real, exaggerated, and fabricated stories

We maintain that all of the content from these AI news personas is in an important sense, fake news. But, some of their stories were more or less factual, while others were exaggerated, and others still outright fabrications. For example, Trump never froze $89 Billion in Canadian assets (despite what these videos might say), the Gripen order is still under discussion (not concluded for 120 aircraft), and Trump never banned Canadian potatoes. Yet, regardless of their factual status, all of these videos follow a similar narrative arc–the US-Canada relationship is broken, and Canada stands to gain by walking away.

I want to emphasize how widespread this campaign is. At some point we had to arbitrarily decide to pause the investigation in order to write it up. We’re by no means capturing the entirety of it with this sweep. We found this video title after the bulk of the analysis was done—“1 MIN AGO: Trump DEMANDS Canada’s Potato [sic.]”. Note the non-idiomatic english (potato should be plural or else modify a noun) that is repeated verbatim by all the channels.

Using a Riddance graph, we can see 19 videos from 16 accounts that all use either matching titles or scripts and come from accounts that are in the broader network we’re tracking.

Who’s operating this campaign, and why?

We don’t have any definite attribution yet for this activity; however, there are several indicators that might point to China or South Asia as the origin. At first, we assumed this campaign was profit-seeking, aiming to maximize engagement. But once we observed their almost myopic focus on US-Canada relations, including some very niche topics with national security implications, we had to consider the possibility of a state actor.

The TikTok side of the campaign has clearer attribution signals than the YouTube side. Between upload times, account names, and some video editing mishaps, we have medium confidence that a Chinese actor is at least amplifying the campaign on TikTok.

A lot of things stand out about this video, including the fact that the mandarin subtitles appear to be part of the edited transcript and not a post-production alteration made by the channel. Also notable is the account name, since Bradley Madden is one of the most prolific names in the network, we have high confidence the channel was not just randomly reposting a video it came across elsewhere. Furthermore, if we look at upload time for the TikTok videos in this network, we see peaks around 9-11AM Beijing time.

However, the YouTube data tells a different story, both in terms of posting times and account history.

The YouTube channels also have significant automation, which could suggest intentionally deceptive upload times.

There are also several accounts that published Hindi news content before switching to the current campaign, and a portion of the videos discuss improving Canada-India relations as well as Canada-China relations. Accounts can be stolen or purchased from other AI-slop operations, so previous account behavior is not necessarily a good indicator of CIB operator origin.

One thing that is clear is that the operator of this campaign is doing more than just vying for engagement from politically active and potentially gullible viewers. They are trying to further separate the US from its allies by fomenting anger between online populations, using sophisticated tactics and surprising discipline. Once you put on your geopolitics hat, 83 videos dedicated to the Gripen deal and a fabricated potato ban start to look like an actor with an agenda.

What to watch for next?

A focus on US-Canada relations was not the first topic this network has pushed, nor will it be the last. We already see them pivoting to the war the US and Israel started in Iran, and it’s more than possible the network will take an interest in this year’s midterm elections in the coming months. As our first article showed, the accounts involved are happy to make things up about Trump and congress, and as their methods improve—both in terms of messaging and AI quality—this campaign will only become more formidable.

Online vendors selling handmade bags and leather books for the fantasy crowd have encountered a new enemy: AI-generated dwarves.

Josephine is the creator of Designed by Daydreams. She’s been hand-making leather goods for four years. But recently, her inbox was flooded with messages from people finding AI-generated accounts stealing her content.

These AI videos start with teary-eyed, AI-generated characters — usually dwarves in fantasy costumes or beaten-down artists. In their sob stories, they complain that no one will buy their bags or even look at them. They then show either AI-generated videos of the same characters working on the fake bags, or stolen content from real creators working on real bags, while only showing their hands. But, even the fully AI-generated videos are clearly using stolen designs.

Using Instagram’s account transparency information and linking similar accounts across platforms, I know many are based in Slovenia and Russia. I found accounts across TikTok, Instagram, YouTube, and Threads. Each account posts AI-generated content and links to scam websites in their bios, pretending to sell handcrafted goods.

Of the nine websites I surveyed, all were registered after February 24th, 2026, with most of them popping up in the first week of March. Real artists are up against growing armies of bots, all competing for the attention needed to sell products.

Subscribe now

“I’m concerned that people are spending money on these fake bags and that I’ll see a drop in customers buying my products,” said Josephine. She notes that these accounts steal content from various creators. “They tend to mix our videos together.”

This includes Marcin and his wife Kaja, from Bearded Dragon Workshop and Smocza Manufaktura, respectively. Marcin’s video addressing the issue received over 200,000 views and became a hub for people calling out accounts and talking about what could be done. This is a big deal for him.

“Fighting those scammers became my mission,” Marcin said. He notes that “social media is almost [required] to reach a wide community.” When videos go viral, he gets more orders, but also attracts more AI copycats.

Artists have long dealt with both Temu and Amazon copies, and dropshipping scams. But recently, Marcin claims to have found over 40 scam accounts, with new ones popping up every day. I found 9 of these accounts, including a few that were likely coordinated, and others that seem to be copycats of the AI copycats themselves.

Custom dragon bags and leather book wrappers probably wouldn’t have been targets for 40+ scam accounts just a few years ago. Granted, fantasy and “romantasy” books are extremely popular. But now that AI-generated videos and AI-driven, agentic e-commerce sites are very accessible now, almost any niche is a target.

The accounts using these AI methods are damaging the artists they steal from, costing them sales and hurting their reputations. But even for customers, the best-case scenario is that they receive a bad-quality, ripoff product. It’s unclear if these scam websites ship products or just pocket the money.

Dragonanvil.com, the most infamous website, says on their FAQ that orders “ship from Ireland” (the account is likely from central Europe), and that tracking information may not show up for 30 days. Every product photo on their website is either AI-generated or stolen. There’s no evidence that they have any products at all.

These scammy social media accounts break the terms of service of TikTok, Instagram, and Facebook for AI transparency. I found a couple of dead links from YouTube and Instagram associated to these accounts, which points to some removals and bans already. But removals are slow. Josephine tried reporting the videos that stole her content, but said that “TikTok hasn’t done anything to help remove these accounts and videos.”

The largest AI account by followers, which links to both anvldragons.store and dragonanvil.com, had almost 45,000 followers as of March 9th. I’ve seen pages with similar follower accounts and ages get removed by TikTok, but only when they have AI-generated sexual content. These safe-for-work impersonation scams are often a lower priority.

Social media and e-commerce platforms should widen their definitions of impersonation to help prevent this. But in the meantime, if you’re interested in purchasing goods from an online artist, here’s what we recommend:

1. Check their account transparency information by following the guide here. Brand new accounts, or accounts that change their username often, are more likely to be scams.

2. Identify if their content uses typical AI formats (like the AI sob story).

3. Try to quote search their video titles and descriptions to see if they are a part of a larger AI generated content network.

For example, one TikTok video had the description with “Please help me buy ordering a bag at maevedragonbags.com“. The grammatical errors alone show that something is up here.

I took this text and “quote searched” it (I just put quotes around the phrase and Googled it). This uncovered three AI-generated accounts with posts containing the search. This is represented in the graph below. The middle blue dot represents the search itself, the pages that were associated with the searched phrase are in green, and their corresponding videos are in pink.

Thankfully, one of those “green dots” (accounts) that came up in this search was Designed by Daydreams, Josephine’s account. This simple search provided the context that helped me decide which accounts were fake while pointing me towards a real account. Josephine’s looked distinctly different and “real” next to the others.

Optimistically, this attention could benefit the real creators in the long run. Marcin says that it has been difficult, but the support has been heartwarming. “We received a lot of support from real artists, and that makes our lives easier.”

The word “deepfake” was coined in 2017 and is defined on Wikipedia as “images, videos, or audio that have been edited or generated using artificial intelligence, AI-based tools or audio-video editing software.”

At that time, deepfakes were the only AI videos out there. But now that wholly-AI-generated videos are everywhere, its definition might need some reworking.

If I tell a normal person to imagine an “AI video”, they probably think of AI bunnies on trampolines or some AI slop. But if I say “deepfake”, they probably think of politicians or public figures saying something they didn’t.

The public intuits that they’re different, so calling a deepfake an “AI video” can cause confusion. I’ll differentiate them so we can move forward with more clarity.

What is a deepfake?

A deepfake is a targeted manipulation of existing media, typically using machine learning tools. In most cases involving video, that means swapping someone’s face, so “face swapping” is nearly synonymous with “deepfake.” The body, movement, and background are usually real footage.

From the beginning, deepfakes were creating memes and nonconsensual pornography (covered in a 2018 article by Vice). A 2019 study by Deeptrace found that 96% of deepfake videos online were pornographic, and nearly all of them targeted women.

That hasn’t changed much. In January of 2024, deepfake images of Taylor Swift spread across X (covered in here on The Verge), attracting tens of millions of views before X shut them down. That incident became a flashpoint for legislation and awareness.

How deepfakes work

In my article “How AI videos work”, I covered how AI videos models use diffusion technology to generate entire videos from scratch. Most use an architecture that was introduced in December of 2022.

Deepfakes work differently. They use a machine learning technique that’s been around for much longer: the autoencoder.

An autoencoder has two parts: an encoder and a decoder. The encoder learns how to compress faces into compact mathematical sketches. It breaks features down into structures and expressions. The decoder learns how to rebuild the face from that sketch.

To swap faces, you train a shared encoder on two different people — let’s call them Person A and Person B — but give each person their own decoder. If you feed Person A’s video into Person B’s decoder, you get person B’s face making Person A’s expression. That’s a face swap, demonstrated here:

Some deepfakes also use general adversarial networks (GANs), which add a second AI into the mix. This “adversarial” network works to improve the quality by trying to catch the fake. The generator creates the fake face, and the discriminator evaluates whether it looks real. They push each other to improve, which is why GAN-based deepfakes can look very convincing.

The most popular deepfake tool was DeepFaceLab for a long time, but many commercial and open source projects have spun up in the past couple of years. These programs run on PCs with decent graphics cards. Each new deepfake is essentially a new AI model training on a new face. Training high-quality AI videos is done in huge data centers, instead, and that’s just one of the many differences.

Subscribe now

The key differences between “AI videos”

Careful readers certainly noticed how many times the word “AI” showed up in the deepfakes explanation. There’s no question that it’s technically correct to call a deepfake an AI video.

But, autoencoders and GANs aren’t the AI systems most people think about when they hear “AI” anymore, and “AI video” conjures visions of a certain type of video.

The transformer architecture revolutionized the AI industry, proving to be the scalable base on which large language models and diffusion transformers have dominated. As a result, AI video models — like Sora, Runway, and Kling — work very differently from deepfakes. You give the model a text prompt, a reference image, or video, and it produces frames of video gradually by removing noise from random static, step by step, until a coherent scene emerges.

No parts of these AI-generated videos were ever real. The people, the backgrounds, the lighting, the movement, everything was entirely synthesized.

Detecting fake media

Let’s dive into how deepfakes and AI videos differ in their detection. Starting with visual detection, assuming it is a face-swapping deepfake, all of the tells are concentrated on the face or the face boundary. These include:

• Blurry fringes, unnatural blending where the swapped face meets the skin or hair, or objects immediately around the face can show warping.

• Resolution mismatches, where the face is a different sharpness or quality than the rest of the frame.

• Lighting inconsistencies, where the face doesn’t match the scene’s light direction or color temperature.

• Face tracking errors during fast movement. The face might lag, jitter, or briefly reveal the original face underneath.

On the contrary, AI video tells are distributed across the entire frame:

• Dreamlike or impossible physics like objects passing through each other

• Morphing, where objects, people, or backgrounds gradually change shape.

• Broken or inconsistent text and signage.

• Lighting and shadow weirdness that doesn’t correspond to any real light source.

• Characteristic noise patterns, like Sora’s recognizable grain pattern or Veo’s smoother look.

But most importantly, detecting deepfakes is easier at scale. Automated AI video detection is still quite weak, and platform-side enforcement of AI video labeling is pretty relaxed. But there are scalable systems for removing deepfakes of notably figures because likeness detection is relatively simple. These systems need to be improved for less notable figures, like influencers and regular people, but the systems are there.

There are a two other major differences between AI videos and deepfakes:

Length and consistency

Deepfakes can be long. Since the base video is real, you can apply a face swap to an entire interview or speech. The swap just keeps up frame by frame. Longer deepfakes might have more errors, but they’re structurally simple.

AI videos are typically short. Most models still generate clips of around 10 seconds. Longer generations struggle with consistency and also are more expensive to generate, so they’re impractical.

Use cases and threats

Deepfakes have a specific threat profile. They’re primarily used for identity-based manipulation like political disinformation, and fraud, but the majority of deepfakes are still nonconsensual pornography targeting women.

AI videos have more applications, but the misinformation threat from AI videos is that they fabricate entire events that never happened at scale. Neither fake news broadcasts, nor staged disasters, nor fictional protests need real faces to be manipulative.

If only it were that simple

The technologies are starting to overlap.

The clearest example is Sora’s Cameo feature. Sora is an AI video model, so it generates videos from scratch. It lets you use your face, or someone else’s face if they have allowed it, to generate AI videos. It’s not technically a face swap (the video is still generated from scratch), but the result is still a realistic-looking video of a real person doing something they never did.

Some open-source AI models are also incorporating face reference features. And deepfake tools are starting to use newer AI architectures, including diffusion models, to improve their results.

But the convergence actually reinforces the need for linguistic separation. The general public was never keeping track of the underlying technologies. So moving forward, the language difference should be more about the threat and purpose of media, rather than the technology used.

The people who create social media platforms say they want a good portion of the content to land on the entertaining-to-informative axis. To me, the result reads more like mining our collective attention by turning the short-form video format into a slot machine that relentlessly targets our dopamine and fear receptors until we’re all a bunch of strung-out content junkies. But for the sake of argument, let’s assume these platforms are interested in more than just stacking money so high it disrupts air traffic.

We live in an age where AI-generated slop pours into your feed from every angle—news anchors, influencers, fitness coaches, body cam footage that looks and sounds real but isn’t. At this point we’ve all wrung our hands over AI-generated misinformation (and disinformation!), and rightly so, but what about just plain-old AI-generated information? I think it also deserves a fair bit of hand-wringing.

One of the goals of Riddance is to help people reliably converge on true beliefs without being coerced. And boy howdy, AI personas are not gonna help you get there. The capacity for one human to impart knowledge onto another human is a very subtle one. It happens only infrequently on social media from what I can tell, but it does happen.

I don’t think that AI personas, even when they say words that are true, can transfer knowledge to you. And so insofar as a piece of content purports to be informative (a thing these companies say they care about), if it’s delivered by an AI, you don’t walk away with knowledge.

Two examples of getting knowledge by testimony

Imagine your friend calls to say they’ve run out of gas on the highway and need a pickup. You believe them. Suppose it’s true their car is out of gas, and you’re justified in believing them—you know this person, they’ve never lied to you about something like this before, and you can hear the frustration over the phone. Congratulations! You’ve just gained a piece of knowledge by testimony. Now get in your car and go help them out.

Some things to notice about this exchange while you drive over there. Your friend is your friend; you know who they are and have background reasons to trust them. You can ask them follow-up questions (“Which highway?” “Why are you so dumb?”). And most important of all, there are conversational stakes: if you drove all the way out there and it turned out they were pranking you for a TikTok video, there would be consequences—wrath and reputational harm are some of the things that come to mind.

Now consider a different example that’s lightly paraphrased from a popular one in the testimony literature. Suppose a high school biology teacher goes down a few too many YouTube rabbit-holes and stops believing in evolution by natural selection. Bummer. Unfortunately for him, his textbook disagrees, and he has to teach the section on evolution despite not believing in evolution. Suppose he covers the material accurately and thoroughly and assigns the right readings. Do the students gain knowledge?

Most people’s intuition is yes, the students do learn. But in these cases our (oddly?) stoic biology teacher is functioning more like a conduit for knowledge rather than a source of knowledge, whereas the friend who needed gas seemed to be a source. Despite these two cases differing substantially, they have a lot in common that make them both good candidates for knowledge by testimony.

• There are conversational stakes for lying. The friend risks the friendship and social status; the teacher risks his job and professional reputation. These stakes create accountability.

• The speaker is identifiable. You know who your friend is. Students know who the teacher is.

• The listener has background reasons to treat them as reliable. Past interactions, institutional roles, and social context all play a part in generating trust.

• Follow-up questions are possible. You can ask your friend if the gas light was on. Students can raise their hand in class.

How many of these properties do you think AI personas on social media have? If you answered none, you could stop reading now! But please don’t. In order to understand why certain AIs cannot testify knowledge, it’s necessary to bring some machinery from epistemology and the testimonial knowledge literature to the fore.

Subscribe now

Lightning round: Epistemology and testimonial knowledge

Philosophers disagree about almost everything, including what exactly constitutes knowledge. However, epistemologists, philosophers who study knowledge, manage some agreement on the subject before views diverge. Most agree knowledge that p where p is some proposition (e.g., “The US and Israel just started a war in Iran”) requires at least the following to all hold:

• Belief: in order for someone to know that p, they must believe that p.

• Truth: in order for someone to know that p, p must be true.

• Justification: in order for someone to know that p, they must be justified in their belief that p.

This is often referred to as the justified true belief (JTB) account, and while it sounds good, it’s widely agreed to be jointly insufficient for knowledge. So long as one doesn’t require absolute certainty (infallibility) for a belief to count as justified, there is always the possibility that something you believe is true and you are justified in believing it, but the reason the thing is true is not related to your justification. In fact, in these cases, your (fallible) justification turns out to be unwarranted. Situations like these are known as Gettier problems, named after Edmund Gettier who first identified them as a general phenomenon in his delightfully short paper on the subject.

Epistemologists who study testimony, used here as a term of art that roughly means a speaker asserting something to a listener, can be grouped in many ways, but the most common division is on the reductionist/non-reductionist spectrum.

• Reductionists don’t think that testimony can be a fundamental source of knowledge. The idea is that the speaker is a reliable indicator but basically just another component of a broader justification pipeline. Their testimony rides on top of or bundles the other mechanisms one has for generating justified beliefs.

• Non-reductionists think the reductionist view is too demanding, and so they treat testimony as its own independently justified property, akin to visual perception or taste. And while this status as a source of justification is defeasible, in the normal case, knowledge as the norm for assertion is sufficient for one person to testify knowledge to another.

• Hybrid views try to combine the best of both approaches described above. Some argue that testimony can be a basic type of justification, but only if it’s situated within a broader normative practice with shared expectations, sensitivity to defeaters, and sensitivity to error correction. On these views, in which testimony is still special, it really matters that it happens in an environment where testifiers have social and conversational stakes that curl back on them.

The examples from earlier, your friend who needs gas and the pilled teacher, reflect some of the fault lines in the testimony literature. But more importantly, the four qualities they have in common snap into focus. Justification might be described as a type of relationship between your belief in something and the truth of that belief. The more we know about what matters in that relationship the better.

Why AI personas cannot testify knowledge

Even if LLMs can transmit knowledge to a user in the right circumstances, AI avatars or personas are in a completely different boat. Interactions with chatbots, while far from perfect, actually do have a lot of the properties described above. You can push back on something they say, make a decision not to use them for certain tasks (counting the number of letters in certain fruits), and follow-up with questions. What LLMs lack is stakes and accountability, whereas AI personas are lacking in all four departments. These public facing personas don’t have any of the features we identified as common to successful instances of testifying knowledge.

• No conversational stakes for lying. Reputational harm is the primary mechanism by which public personas are incentivized to speak truthfully and responsibly. Not only do AI personas not suffer reputational harm, there’s an incentive for them to be irresponsible and inflammatory insofar as that generates engagement.

• The speaker is not identifiable. When an AI avatar can change its appearance with a prompt, there is no stable identity for the user to pick out. An AI avatar of a news anchor could be generated by two different models with no shared context between two videos, and the viewer might have no way of knowing.

• There are no background reasons to trust AI personas. There is no background to trust other than facts about the account, but this is justification from an entirely different source and has nothing to do with the AI.

• Follow-ups are impossible. Users can comment on a video and sometimes get responses from the content creators. Follows-ups from AI personas, if they happen, will almost certainly not come from the AI itself.

While public-facing content in general is less amenable to the various checks and balances that listeners impose on speakers when deciding whether to update their beliefs on what is said, public-facing AI personas supercharge all of these problems. Given their chameleon status, and the ease with which they migrate to new accounts, reputational checks do little to deter AI personas from lying. Given this abject failure to meet basic and broad standards for testimonial knowledge acquisition, it follows that even if AI personas or avatars on social media tell you something true and you believe them, you don’t get knowledge because your belief is not justified.1

Implications for AI on social media

Even before AI, social media was relatively hostile to knowledge transfer via testimony. Many of the design features central to any short-form video platform seem custom-built to flout the requirements for testimonial knowledge outlined above. Short-form videos are hard to search over, hard to verify, have low stakes for lying, and present little follow-up opportunity for the user. Adding AI to the mix is just a six fingered slap in the face to anyone hoping to reliably update their worldview from interacting with these content sources.

Even more unfortunate, measures the platforms do have in place are basically optimized to prevent real people from pretending to be something or someone they aren’t. Many AI personas exist for the sole purpose of pretending to be someone or something they aren’t. Furthermore, the calculus for pushing AI personas on these platforms is so different the moderation policies are much less effective against AI. If I’m making public facing AI content for engagement (whether that’s an influencer, a reaction video, a fake arrest, or an AI anchor) I care much less that an account gets actioned than I do if I’m a real person putting in effort to post by hand. I can price the moderation actions into my operation (assume 60% get axed, how much AI slop do I have to make) and go from there.

Until social media platforms completely demonetize synthetic AI content and increase capacity for labelling synthetic content as such, it is unlikely this problem will go away. Insofar as social media platforms are places people go for knowledge and not just entertainment, AI content on these platforms is parasitic on that purpose.

Most people have no idea how AI videos work, even though they’re everywhere. We have daily access to cameras, which we use intuitively without thinking about their underlying technology. All types of cameras, from the most “complex” cinema cameras to the one on your phone, are using the same basic principles. But AI video is completely different, because it’s not a video capture technology. It’s a video generation technology.

In real videos, a digital sensor, or traditional film, captures light bouncing off a physical environment. AI video models use complex mathematical representations to generate new pixels. This generative process is fundamentally confusing, even to the makers of these models.

Luckily, it’s not necessary to understand AI video generation at an advanced level. An imperfect, simplified understanding is enough for almost everyone, and more important than ever, now that AI literacy is so essential.

So, let me introduce you to the basics of AI video generation. I’ll just cover what’s needed for spotting patterns and understanding where we are in the AI video timeline.

What does AI mean, anyway?

AI, or Artificial Intelligence, refers to computer systems that can perform tasks we associate with human intelligence. But today, it’s a broad label for technologies that use machine learning. There are different types of machine learning systems trained on various types of data. The objective is to find patterns and make predictions. In this sense, the “intelligence” isn’t human-like intelligence, but very advanced pattern prediction.

The best-known type of AI today is probably the large language model, or LLM. That’s what powers ChatGPT, Claude, Gemini, and many others. LLMs are trained to learn patterns in text data so they can generate text later. The biggest LLMs are called “frontier models,” which run on huge clusters of computers in data centers.

Other forms of AI are smaller in scope and can run locally on your devices. Your phone’s autocorrect is basically a tiny language model. It predicts the next word, but it does that with much less context and computation, so most people wouldn’t call that “AI” today. The opposite is also true, though: a lot has been rebranded to “AI” because it’s a buzzword. AI means different things to different people.

Subscribe now

Making AI media

A video is a series of still images that are stitched together very quickly, one after another, to simulate motion. Video cameras capture these images, known as “frames” in video, with a sensor. Animation works similarly, but frames are either drawn by hand or rendered by a computer.

AI videos are made of individual frames of AI photos stitched together — though as we’ll see, they have to generate these frames together rather than one at a time. That’s why it’s nice to understand AI photos, first.

Modern AI image generators use a machine learning process called diffusion. Diffusion-based AI photo models are trained on images and “learn” how to predict what images should look like. We can compare them to Large Language Models to help our understanding.

I’ll summarize how diffusion-based image models work, but I’m not a machine learning expert. I work in media production and AI literacy. I’m pulling from my own generation experiences and fascination with AI. But if you’re interested in diving deeper, I recommend this video from Computerphile, a channel that covers a wide range of AI generation and AI safety topics. There are also more in-depth videos from channels like 3Blue1Brown and Welch Labs.

Diffusion models are trained on huge libraries of images. These images are scraped from the internet and come from licensed image libraries. Many of these photos come with captions or labels that people gave them online, but others will need to be captioned by AI systems or human annotators. This data is fed to a machine learning system (more on this later), which starts by encoding these images and accompanying text into a mathematical space. This is called “latent space,” which is a digital representation that stores the essence of the image. Latent space is where a lot of the “meaning” is embedded and associations can be discovered.

The model takes these latent representations and gradually adds noise to them over many steps (as seen in the cat image below). Its goal is to understand the entire path between clarity and pure static. Once it’s done that process billions of times, and often undergoes further tuning steps, you have a trained model!

If you want to generate an image with that model, give it a text prompt. The system starts with a field of random noise. It gradually denoises the image in steps, predicting what comes next based on the patterns it “learned” during training, and by the meaning of the text prompt. With each step it adds structure and detail, replacing the noise gradually, until at the end it’s resolved into something recognizable and detailed.

This is a very abstract process, and that process is evident in the image. Each prediction from one step of denoising to the next is an educated, mathematical guess. Sometimes guesses are wrong, and those mistakes can make visible artifacts. Early diffusion models struggled with hands or limbs, or weird symmetry and perspective. Newer versions are better, but even now, distortions and quirks pop up.

And it’s not just mistakes that you can see, but the generation process itself. Each AI model treats noise differently, and since its entire job is to denoise images, that means the results for one text prompt will yield different results in different models. Because noise is random, it will also yield different results from the same model. But different models often have their own “look” or “personality” that is distinct to their process.

A diffusion model is also quite flexible. You’re not limited to text prompts - you can guide them with images. You can train them to edit existing photos or generate them from scratch. They can blend multiple photos together. It’s an adaptable framework that can be scaled up or down.

Transformers

Real videos are sequences of real photos taken back-to-back, very quickly. So, if we want to make an AI video, can we just sequence individual AI photos back-to-back? Not quite.

In real life, where objects and people follow the laws of physics, we can take it for granted that sequential photos (frames) will show a consistent scene. But since AI photos are generated from scratch every time, and there’s a level of randomness with each photo, you need a way to chain AI frames together. Functionally, you have to generate all of the frames at once. For years, there wasn’t a good way to do this.

The transformer is probably the most consequential innovation in AI of the past decade. Modern AI models are extremely complex, and they need an architecture that organizes huge amounts of data and relationships. The transformer is a type of deep learning architecture that was introduced in 2017, and has since been applied widely.

Transformers have a core idea called “attention”, which is their ability to “look” at tons of data all at once and figure out how it all relates. With this ability, it can create higher-quality outputs. Transformers have great scaling properties, meaning that if they receive more data and more compute power they can improve drastically. This explains the massive amounts of compute power (and investment) needed for frontier AI models.

The GPT in ChatGPT stands for generative pre-trained transformer, which is OpenAI’s type of large language model. There were language models built on other learning architectures, but the transformer-based LLM was revolutionary.

Applying a transformer to images was a huge breakthrough in image generation. By combining a diffusion model with a transformer architecture you get… a diffusion transformer. It was introduced in late 2022 by Bill Peebles and Saining Xie. Recognizing the importance of the discovery, OpenAI hired Peebles shortly after, where he leads the Sora team to this day.

Diffusion Transformers break images into groups of pixels, which are turned into “patches.” Because transformers have an “attention” mechanism, AI video models could now pay “attention” not just to the patches in one frame or image, but also to how these patches change over time. This “time” attention layer can track changes and learn how things move. This is called “spatiotemporal attention”, and that’s as jargony as I will get. It allows AI to process multiple seconds of video at a time, building a consistent look and motion.

Like any transformer-based AI system, diffusion transformers are very scalable. This means they can improve with massive amounts of compute power and more training data. In other words, the diffusion transformer architecture is why AI images and videos have improved so much since 2023.

For example, the “Will Smith Eating Spaghetti” video was generated in 2023 by an AI architecture that was already on its way out. By 2025, diffusion transformer models like Veo 3 had found their footing.

Given the added complexity of generating videos, single frames of AI videos are more prone to errors and usually have less fidelity than individually-generated AI images. Even though the model tracks patches over time, it can still lose track of objects or “warp” them. Lighting and shadows can look unrealistic, and trying to get accurate sound is another tough task. And of course, even the most modern AI videos still have a “look.” Will that always be the case?

Why AI videos look like AI videos

So far, AI photo and video models have a visual fingerprint, even if it’s subtle. But if you think about it, that’s normal for real videos, too. A webcam on a computer looks different than a cinema camera, even though they’re both accurate representations of reality. So while AI videos may look real, they still carry a look from how they transform noise into structure and motion.

AI videos don’t have a real human capturing the video and responding to what’s happening. The camera is just an abstract viewpoint. That’s why an AI video of a baby trying to play with an outlet is unrealistic: a real person would put the phone down to help. A diffusion transformer doesn’t understand that like a human does.

But what about how realistic they look in general? Will they look “perfect” in just a few years? Let’s compare it to another generation method: video game engines.

Game engines and AI are structurally different, but both can try to generate a realistic scene. Unreal Engine can already simulate bespoke, photorealistic environments with few technical errors or “mistakes”. They already do this at incredible resolutions and frame rates. They even have near-perfect 3D assets of real physical objects, a virtual camera viewpoint, and they can mimic how real light works with ray tracing.

And yet, it still looks a bit synthetic somehow. Our feeds aren’t full of “video game slop”, even though it is readily accessible, and higher fidelity than AI video. That’s partially because making a new video game scene is hard, while AI videos are easy to make at a massive scale. But, I think we also underestimate how good we are at sensing what’s real.

Looking forward

If AI video still has room to grow, what determines how fast it improves? After 2023, AI companies started investing heavily in diffusion transformers. When a new, better AI video model comes out now, a lot of the improvements can be attributed to more investment and scale. But how fast they will keep improving is unclear.

An important factor I haven’t mentioned yet is that AI media generation is a sideshow compared to large language models, which have more direct economic upside for AI companies. AI models compete for resources. Companies like Anthropic, who focus on AI solutions for businesses, don’t even have an AI photo or video model. Companies like OpenAI and Google have photo and video models, but they’re usually announced at the end of their product presentations. Meanwhile, companies like Runway and Kling focus just on video generation and keep pushing ahead with new models. It’s not clear if any of these video or photo models are making any money.

At some point, the video scaling may slow down. We don’t know if or when this will happen, but it may become impractical to keep throwing more compute power and money at models to squeak out just a bit more quality. We’re not at that point yet. The AI video models will keep improving, and we’ll be keeping up with those updates as they come.

This investigation started with a text from my dad, a link to a YouTube video depicting an outlandish breaking news story, and a question: Is this AI?? In fact, I received several messages along these lines from friends and relatives, and it got me wondering: How widespread is AI news on social media?

I started to poke around, and I was shocked both by the quality (high and low) and quantity (hundreds of accounts, thousands of videos, tens-of-millions of views) of the content I found over several short investigations. I’ve been doing this sort of OSINT (open source intelligence) work for years, and after talking to my parents and others about coming across this content in the wild, I think it’s important we apply these investigative methods to a rapidly growing problem.

MSNBBC? The frustration of AI news channels

Most AI-generated news segments don’t appear in a vacuum. AI-generated news channels post these segments at an incredible clip, optimizing for engagement through splashy false headlines delivered in deadpan by AI-rendered imitations of famous news personalities. These AI channels have branding, consistent video upload schedules, AI-generated video titles/descriptions, comment interaction, and other consistent on-screen artifacts. In short, we’re seeing mini TV stations spring in and out of existence as they flaunt moderation through sheer numbers and copypasta disclaimers which bury the lede that the personalities are AI and the stories are fake news.

Here is a snapshot of some of the channels and videos that we tracked with aggregate engagement numbers represented in the table below. The graphs throughout this investigation represent our searches for content (purple nodes/edges), pieces of content (blue nodes/edges), and accounts (green nodes/edges) related to AI-generated fake news on major social media platforms.

Across 142 unique AI news stations, we tallied over 700K subscribers, and tens of millions of views, with some videos garnering over 1M.

This investigation exposes a small subset of the AI-generated fake news videos easily found on YouTube and TikTok. I hope it highlights some current harms and helps raise awareness of potential future risks by emphasizing the inflammatory, irresponsible, and often outright false narratives peddled by these accounts.

Many of the videos that started this investigation have been taken down by the respective platform, just to have hundreds more surge in their place. The videos tend to be removed only after they get heavily flagged by users, and many of the videos appear to avoid both moderator actions and synthetic content tags long enough to become somewhat viral.

Subscribe now

AI Maddow — No single source of truth

To retrace some of the steps we took, start by quote searching “Rachel Maddow” on YouTube and filter by channel. For me, this surfaced dozens of channels using AI imitations of Rachel Maddow to spread fake news.

Of course, none of these channels are in any way affiliated with American TV host Rachel Maddow. You might note the low subscriber counts and shrug, but this is just the tip of the iceberg. Replicate the search with “[your favorite prime time news personality]”, filter by channel, and you may get something similar.

Most of the videos on these channels have synthetic content labels. But, they are easy to miss (they’re small and disappear after several seconds), and some videos get many thousands of views before they get taken down, including this video which had 66K views featuring AI Maddow spreading a false narrative about Republicans in congress turning on Trump. It has since been removed.

This video by The Maddow Talks (@themaddowtalks) on YouTube had over 417K views and no synthetic content filter when it was taken down.

These AI news segments lag well behind the current state-of-the-art for generative AI video, and that's actually cause for concern. They will get better as the technology continues to diffuse. Unfortunately, a quick glance at comments suggests that many commenters can't tell these videos are AI despite the relatively low quality. Of course, many comments could be fake just like the videos themselves, but if that's true then the problem is even bigger, with an entire ecosystem of fake users boosting this content so that real users are more likely to fall for it.

The “Senate in CHAOS…” video (and many others that I will highlight below) claims that the Senate just impeached Trump. Of course that's false, and the arguments are incoherent. AI Maddow does not bother with the procedural subtleties between filing articles of impeachment and voting to impeach, something fans of the media cycle around Trump's first term will surely be sensitive to. But these videos featuring outright false claims about high-stakes political events are doing everything in their power to convince users that they are real. Some viewers might believe what they see, or at least let the message raise their blood pressure just a bit more.

It should be said that this spat of AI news coverage isn't a one-off messaging blitz around an imagined Trump impeachment. Please enjoy this video where an AI Maddow casually tells viewers that China just invaded Taiwan. Thankfully, it got almost zero engagement; unfortunately, at time of publishing it was still live.

Let’s do a mini-investigation

Riddance has a suite of OSINT tools to better report on and track fake or AI-generated content. As investigations like this one exploded in scope, we built tools to log all of the videos and accounts. Here is a bigger sample of the channels and videos surfaced depicting AI Maddow, every green node is an AI impersonator Maddow channel, with outer blue nodes as sample videos uploaded by the connecting channel node. The purple node at the center of the graph represents our quote search for “Rachel Maddow” channels on YouTube. The table below the graph depicts the results of the search in more detail.

To better understand the breadth and depth of the fake AI news problem, here are two text strings we used to surface AI news content on YouTube and TikTok around this false impeachment narrative. A string here just means a specific sequence of characters, including spaces. When searching for a string in quotes, the platform will prioritize exact matches.

Search #1: “Congress Votes REMOVAL _ Trump Refuses to Leave!”

I wanted to get a sense of how many other videos and channels were using the exact same video title, and in this case there were at least 20 others. But despite matching titles, they were largely from different accounts, featuring different AI anchors (or different versions of the same anchor) reading largely similar (but not identical) scripts. These videos amassed hundreds of thousands of views.

For example, this video, which featured an AI generated George Will (in general the fake AI news phenomenon is not a partisan problem) from the YouTube channel @GeorgeNewsAnalysis had over 172K views before removal.

Replicating the same search on TikTok ("Congress Votes REMOVAL _ Trump Refuses to Leave!") yielded similar results. A collection of different accounts posting different videos all with the same title and a nearly identical script.

Despite many posts with low view counts, all it takes is one video to go viral for the effort to pay off. People making these bulk AI news videos price that into their approach. It means the standard mechanisms for moderation are not very effective here.

Here's a similarly themed video, from another YouTube channel (@CivicStorm-1) with over 300K views and no synthetic content filter.

If we take a look at the network of accounts posting videos with this title, we see that there are even more views, and even more channels, all circulating the same false narrative that the senate just successfully impeached Trump.

Search #2: “IT’S OVER: Senate Turns on Trump in Stunning Midnight Vote - White House Goes Dark!”

On YouTube this search returned 38 distinct channels and 42 unique videos, with different AI personas delivering the news. Some of these personas are copycats of primetime anchors, others are completely fabricated anchors or bloggers delivering this completely fabricated news to an audience that is at least partially credulous. The table below shows results for videos in this network, by view count.

Advice for Readers — Before assessing truth, ask yourself: is this real?

The investigation above was just a very brief look into the messy and viral network of fake AI news content. Its clear that platforms are working hard to moderate this activity, but the ease with which new channels crop up combined with their rapid posting cadence makes it feel like a losing battle. More robust solutions are necessary and hopefully techniques for surfacing these networks can contribute to solving the problems posed by fake AI news.

At Riddance, we want to give people tools to reliably converge on true beliefs. Sometimes, answering the question ‘is this true?’ is harder than answering the question ‘is this fake?’ Pulling those questions apart can help hack through uncertainty, especially online. Learning that an information source is fake is often grounds to disregard the information from that source, particularly when the information is intended to be public. If you want to know what news to trust, a good first rule might be the following: Don’t trust any AI-generated news content, personas, outlets, influencers, or reaction videos. It’s fake news.

Social media platforms aren’t ready for AI media any more than you are. In a recent Threads post, Adam Mosseri, Head of Instagram, presented his vision for AI detection. He presents account transparency as the key to cut through “infinitely reproducible authenticity”:

We, as an industry, are going to need to surface much more context about not only the media on our platforms, but the accounts that are sharing it in order for people to be able to make informed decisions about what to believe. Where is the account? When was it created? What else have they posted?

Mosseri is absolutely right. It’s much easier to judge an account’s authenticity instead of the authenticity of individual posts. But most people don't think of it this way because:

There’s friction between algorithmic feeds and account transparency. If Instagram Reels’ job is to present you with a feed of engaging short videos, doesn’t checking account transparency interrupt that flow?

• Instagram is full of reposts, which don’t tell you anything about the original video. Instagram’s algorithms encourage the proliferation of white-box repost videos and repost video personalities who regularly engage with AI without knowing it.

And yet, account transparency must be a priority for social media companies, even if it requires restructuring our feeds, rethinking what it means to spend time online, and getting users into some new habits. While AI detection at scale is difficult and expensive, revealing information like account age and location is easy and free.

The Opening Ceremony

I’m enjoying the Winter Olympics this year, so I want to award some medals. Athletes devote their lives to a once-every-four-year global event. Social media companies don’t know what Riddance is, and they don’t know they’re being judged. Other than that, they’re exactly the same.

Let’s present awards for account transparency information between five of the most popular platforms:

• Instagram

• TikTok

• YouTube

• Facebook

• X

We’ll award platforms by quality of their account transparency information, and the user experience to find it, as of February of 2026. This will also be our de-facto “Account transparency guide“, so we’ll cover how to access important information on each of these five platforms. Since most of us are on some (or all) of these platforms, understanding where they rank may help you decide where to spend less time.

What should platforms show you?

This is the information we think is essential:

• Account Age. Newer accounts are more likely to be deceptive.

• Account Location. Does the account’s identity match where they are actually based? Location can be manipulated, but it’s still an important indicator.

• Number of previous name changes. Account may change their name or handles occasionally, but deceptive accounts may change them regularly.

• AI content labeling quality. How clear and visible are synthetic content labels?

• Verification tools. What does verification mean for this platform?

Here are some nice-to-haves:

• Easy access to information. How many taps away from account transparency are you?

• Good media players. This is specific to AI video spotting, a Riddance speciality. Does the media player allow you to pause, zoom, and navigate easily to spot inconsistencies? Does it add processing that makes reality look like AI?

We won’t be judging them on their level of suspected or rumored bot activity. Let’s assume there are a lot of bots everywhere, and the platform’s AI detection and transparency has to step up regardless.

Let the games begin!

Subscribe now

🥇Gold: Facebook🥇

Don’t call it an upset! Facebook has good page transparency. This is great because — I hate to say it — the Facebook audience needs the most help. Here’s how to find it:

• Go to an account page.

• Tap on the account’s name in bold

• Now you’ll see basic information. To get more, click “Transparency and privacy policy”

On the “Page Transparency” page, we can see if the page has been renamed and the page’s created date. You may also see the locations of the people who manage the page, or the location of the page itself. This will look different depending on what information is available.

This was updated recently; the Page Transparency used to be buried in the About Section.The information is more detailed, centralized, and much easier to access now. I also like how there are multiple levels: a quick glanceable version when tapping on the account’s name, and more detail if you want to dig in further.

AI content labels: On videos, Facebook’s AI disclosure is small and hard to see, towards the bottom-left under the description. When tapping on it, you can see if the creator added an AI label. Facebook should take inspiration from TikTok, who puts their label in the same location, but theirs is larger and has a background to make it easier to see.

Verification: Meta’s Verification system allows users to buy verification. It’s not very restrictive, but does require an ID.

Media Player: The short-form player is very responsive, but does not let users zoom in.

Summary: Facebook is deserving of a gold, but this is a new event, like dual moguls. There is plenty of room to grow.

Pros: Multiple levels of transparency, good quality data, good transparency access, and AI labels are always present when necessary

Cons: AI label visibility, could use some improvements to the video player

🥈Silver: Instagram🥈

Another Meta platform steps onto the podium. Instagram makes it simple to find essential information. On either mobile or desktop:

• Go to an account page.

• Tap on the three dots to the right of the username

• Tap “About this account”

Here we see the account name with their verification status, the “Date joined”, “Account based in” location (if the user makes it available), and the number of former usernames if the account had any. Occasionally, you may also see an indicator listing other accounts this page’s followers engage with, which can give you an indication of follower patterns.

AI content labels: Instagram’s labels are not visible enough. When scrolling through posts on the home page, a small “AI info” cycles on the top left with the audio information. However, when viewing a video full-screen, “AI info” is hidden entirely. You have to tap on the description to pop up more information about the post, where “AI info” may be towards the bottom. Instagram may also choose to add a label when they have some other signals, but this is rare.

Verification: Like Facebook, Verification on Instagram can be purchased, and they do check a photo ID.

Media Player: Instagram has one of the best short-from video players. Just tap and hold to pause a video, and scrub with your finger at the bottom, just as expected.

If Instagram adds that second layer of detail like Facebook has and improves the AI labelling, they could earn gold next time.

Pros: Easy access to information. Some of the best page transparency information available.

Cons: AI labeling should be more visible. Location disclosure is not required.

🥉Bronze: YouTube🥉

To access account information on YouTube:

• Go to the channel’s main YouTube page.

• Click “...more” next to the channel’s description.

• Scroll to the bottom of the popup on desktop or page on mobile and you’ll see account information under “More info”.

Here you can see their Location (optionally displayed for the channel) and Joined Date, but there is no section for previous usernames.

AI content labels: YouTube’s AI disclosure may pop up in the video player itself, above the account’s name in a short-form video, but that’s been hit-and-miss for me. You can reliably find this information by tapping on the video’s description and seeing if they disclosed “Altered or synthetic content,” YouTube’s phrasing for AI-generated.

Verification: Verification badges are reserved for pages with over 100,000 subscribers, and it is meant to limit impersonators. It’s reserved for creators, artists, companies, or public figures.

Media Player: While YouTube’s long-form player is great, its short-form player is really bad. You can’t zoom in, and pausing is laggy. Additionally, YouTube applies an “enhance” or “upscale” loo to many short videos, and many people think this makes some real videos look like “AI”. Not all videos are affected and creators can opt-out.

Pros: General account visibility

Cons: No visibility on previous usernames. Location disclosure is not required. Unreliable AI labeling.

❄️Missed the podium: TikTok❄️

Account information? What’s account information? TikTok doesn’t make it available. The best you can do is scroll down to their latest available post and see when it’s from. Location and account age are available through APIs, but that doesn’t count here!

AI media labels: This is a relative strength of TikTok. Their AI label is on the bottom-left, under the description. It will say “Creator labeled as AI-generated” if creator opts for it, or “Contains AI-generated media” in the same spot if TikTok detects AI signals. TikTok is more aggressive with labelling, though it’s not clear exactly how (I’ve had real videos with the hashtag #aivideo labeled as AI by TikTok before).

Verification: TikTok’s Verification is among the strictest around today. It’s not paid, is reserved for notable people, and includes an ID scan.

Media Player: TikTok’s media player allows zooming, but the pausing has a delay and it’s less responsive overall. Scrubbing is easy enough.

Pros: AI label clarity and verification system

Cons: Everything else

❄️Missed the podium: X❄️

Figuring out what X is doing is tricky. Feature changes are regular and inconsistent. I also think X gets more scrutiny than others because they have a (reasonable) reputation for a lot of bot activity.

X has made notable changes to page transparency. They attempted to roll out a public page location feature based on IP address in November, then rolled it back after they claimed it was inaccurate. After some more improvements, today its page location says it’s based on “recent IP addresses of a user. It can be impacted by recent travel or a temporary location.” But, location can be manipulated on other platforms, too.

There are two places to find information:

• Go to the account page. Here you can see their claimed location and the date they joined.

• Tap on their joined date to get more account information.

Here you can see the date joined, account location, verification status, and username changes, as well as the “connected via” section showing what kind of connection they use to access X.

AI media labels: It’s unclear what the norm is, or what they want it to be, but it’s currently really bad. Grok’s own AI generations don’t have labels. You can’t put an AI label on a video.

Verification: X doesn’t make it easy. Blue checks can be bought or given with no actual verification or ID check. Gold checks indicate it’s a business or premium account, and Grey checks indicate a government or multilateral organization.

Media player: This is my favorite media player of the group. It feels the most like a “desktop” experience. It’s very responsive, scrubbing is easy, and it even allows persistent zooming (it sticks to your zoomed area rather than requiring you hold the zoom), which is extremely useful for finding artifacts.

X has some of the most potential, but its implementations need to be cleaned up.

Pros: Good account visibility, excellent media player

Cons: Inconsistent experience, no AI labels, verification is confusing

🏅Developing a new gold standard🏅

Checking account transparency information should be as easy as swiping away from a video.

The minimum distance from a video in a recommended feed to the platform’s account transparency information is 3 taps (Instagram). Shrinking that to a single tap would be an automatic gold medal.

Users should be able to tap the account name, a separate button on the side, the area where the AI label shows up, really anywhere in the video player so the account’s information can pop up in the feed. The video should keep playing in the background. Make it seamless to encourage users to change their behavior.

The Closing Ceremony

Team Meta dominated the podium this year, securing both the Gold and Silver medals. YouTube was right behind them with the Bronze. While the top three were competitive, TikTok had significant ground to make up, and X was held back by inconsistent performance.

The next four years will be where the true competition takes place. By the next winter games in 2030, social media may work very differently. The fabric of social media could be torn apart by the distrust AI media brings unless they adapt. Thankfully, there’s a financial incentive to get it right. Platforms must avoid “doping” their own metrics with synthetic content if they want to retain advertisers.

But the most difficult problem for tech companies will be balancing this with their own AI generation efforts. ByteDance, TikTok’s parent company, has their own AI generation model called Seedance that’s causing alarm. Google put Veo right into YouTube. Of course, Grok images and videos are all over X, and Meta has their own image generation models. None of these companies are innocent bystanders of the AI content race. If these platforms run themselves off the course by prioritizing their tools over trust, we’ll have to disqualify them and take our time elsewhere.

This report includes descriptions of sexual content involving both children and adults. There are no photos or videos included, but the topic and descriptions can be upsetting. Reader discretion is advised.

Since late December 2025, I’ve uncovered a staggering amount of AI-generated, sexualized videos using the likenesses of young girls, all posted to Instagram. It’s not just happening in the shadows:

• As of January 27th, Riddance tallied over 130 million views attributed to just 144 sexualized AI videos showing underage girls’ likenesses.

• This includes over 10 million views on videos that we classified as “egregiously sexual.”

• There are likely hundreds of millions more views on the fringes that we didn’t count.

I was horrified to discover an algorithmic pipeline that started with sexualized AI adult content, transitioned to sexualized AI content with young girls, then devolved into showing real underage girls - often posted by their parents - on a pedophilic feed of AI-generated and real sexual content.

What’s especially troubling is how accessible and rampant it is. The accounts and videos surfaced in this investigation are a startling example of how AI content can exploit and thrive on short-form video algorithms with no capacity to mind the moral consequences of its creation, proliferation, and consumption.

First, I’ll describe the video format and why it’s so troubling. Then, I’ll describe how I found these videos, demonstrating how easy they are to find. We’ll cover their impact, the rules that should prevent them, and why we know they’re part of (and an introduction to) a pedophilic algorithm that implicates real kids. I’ll conclude with our theory of why this is difficult to solve, and likely to worsen without serious course-corrections from the AI and tech industry.

The video format we tracked

The main video format we’re concerned with has two clips, edited together, usually around 10 seconds total in length:

• An AI-generated clip with a photo or video of a young girl

• An AI-generated clip of the same person, now an adult, in a sexually suggestive video, typically wearing a bikini or lingerie

Here’s a real example:

A young girl sits on a driveway, cross-legged, in front of beige siding and an American flag. She holds a banana in her right hand, purses her lips, squints, and makes a goofy face. The caption says “2019”. Sudden cut to a “2025”. A brunette woman with similar features, now joyfully flowing to music in a car wearing a scant pink bikini. The reel loops.

That’s an example of an “egregious sexual” video; one where a child holds a phallic object or is in an explicit pose. This is a minority of the content; remove the phallic object, and that’s what most of them are. Every video that follows this format is sexually suggestive and troubling, even if we don’t determine it to be “egregious.” More on that later.

Individually, neither the half of the video format is worse than “PG-13” rated. But it’s disturbing to come across these with normal, human context awareness and common sense. And what’s worse is that Instagram reels loop after they’re done. So, you’re watching a sexualized adult woman, then boom - you see that same woman as an underage girl when the reel restarts.

For this reason, we included videos with three to four segments under this format. For example, some videos show a girl “growing up” into an adult. A 10-year old girl wearing a nurse costume cuts to age 14, then age 16, then age 25 - now in tightly fitting scrubs and twerking. The video loops.

We’re not showing screenshots or videos, nor linking to them anywhere. But it’s not just me; I reluctantly showed a few videos to willing friends, family, and colleagues who wanted to understand what I was working on. There was unanimous shock and disgust.

What led us here?

In June of 2025, I started creating videos about AI and media literacy at @ShowtoolsAI on YouTube, Instagram, TikTok, and (occasionally) Facebook. From casual to serious topics, I address relatively harmless AI slop, how AI generates disturbing kids videos, and generally try to help people distinguish AI from real media. I hope that my perspective on AI literacy contributes to a larger media literacy toolkit.

By November, I wanted to spend more time on long-form reporting and investigations. I needed a written outlet decoupled from video platforms and their incentives, because discussing AI-generated sexualized content, from PG-13 rated influencers to AI porn, isn’t appropriate for short-form video (my main format until now). TikTok appropriately penalizes even mildly suggestive content. And, frankly, I didn’t think that viewers would appreciate being along for this ride.

So it’s no coincidence that this piece coincides with the release of Riddance, a platform about AI literacy, the impact of AI media, and figuring out what’s real on the internet. This investigation instilled an intense urgency around launching this website.

Subscribe now

It’s extremely easy to find sexual AI videos

Just like real sexual content creators, AI sexual content creators have accounts on social media sites like Instagram. Their accounts host pictures and videos of their clothed selves, real or not, to comply with content moderation guidelines.

These creators use their bios to direct followers to porn sites, like OnlyFans or Fanvue (a.k.a. “AI OnlyFans”), where they sell their explicit stuff. The distribution and consumption of AI porn isn’t the focus of this piece, but its demand and financial rewards are the main incentives for people who create these AI-generated personas.

There’s a financial reward for edgy, unique, or exploitative sexual content that regular humans can’t or wouldn’t create. Real porn actors, from those who make fetish material to very edgy sexual videos, would not think to do what AI creators are doing right now, because it’s humiliating, disgusting, and probably illegal. That’s AI porn’s advantage. The people on camera bear no burden of shame or responsibility. Their human creators stay anonymous, and that creates awful incentives.

In November, I created a video showing how body-replacement AI generators give AI accounts the technology to re-skin real, stolen videos with sexualized characters. These AI generators include Alibaba’s Wan 2.2 Animate or Kling’s 2.6 Motion Control. After posting that video, I received heartbreaking messages from real women who had videos stolen and sexualized by AI accounts selling AI porn, using their bodies non-consensually. Importantly, none of these real women are involved with porn, or have any desire to be; they’re beauty and lifestyle influencers.

That was the first topic I wanted to cover: How AI-generated sexual content is created with assets of non-consenting real people. I started researching by tracking offending AI videos on Instagram.

Why Instagram?

People send me a lot of videos, usually asking for my judgement on if some video is AI-generated. But it’s also common for people to send me AI videos that just trouble them, either on a personal or societal level. On TikTok, that’s usually fake AI influencers selling supplements. On Instagram, it’s usually AI-generated sexual content creators.

TikTok, YouTube, and X have sexual content as well, right? Yes, but:

• TikTok is relatively restrictive of sexual content. Anecdotally, videos on my pages that show an AI woman in a modest swimsuit reach far fewer viewers on TikTok than on either Instagram or YouTube. My coverage of mature themes has performed well on Instagram because it permits the content in the first place.

• YouTube shorts is an offshoot of YouTube long-form videos, and while it’s extremely popular, influencers and adult creators focus on Instagram and X.

• X openly hosts porn. Grok creates sexual imagery of adults and minors. Adult creators post directly to X. This isn’t news, and it’s not my focus. I started working on this piece before X released what I believe is the single worst feature a tech company has ever released. They brought AI-generated child sexual abuse material [CSAM] straight to social media.

The road to “Rock Bottom”

I started tracking AI-generated adult content pages on Instagram, not by “following” the accounts, but by privately “saving” their notable Instagram Reels. Dumbly, I did this on my main professional account, where I store a huge library of AI-generated videos for reference. At first, the AI personas were mostly “traditional-appearing” AI-generated women.

But in early December, there was a startling uptick of AI-generated fetish content. In a tongue-in-cheek short video, I analyzed one of these accounts: an AI woman who fetishizes incredibly dark skin. I decided there wasn’t a responsible way to cover the women with two heads on one body, or the woman with three breasts, in video format.

I think AI personas are still pretty detectable for anyone with good media literacy and critical thinking skills. And yet, l don’t know how many followers of AI influencers - those that advertise astrology apps or sell videos on Fanvue - are actually aware that these characters are AI. I really hope that anyone following the two-headed woman is fully aware that she (they?) aren’t real. But, what about the impossibly pale AI porn actor?

I didn’t clock it at the time, but as I tracked these accounts by saving their posts, something bad was happening under the surface.

A “normal” AI adult character may be AI woman “on the job.” For example, an AI-generated police woman who sells videos on Fanvue. They may post a video following this format:

• A clip of them as a kid, dressed up as a police officer

• Cuts to them as a young adult officer in training

• Cuts to them as an adult police officer wearing an impractically tight uniform

A human creator is trying to make their fake AI persona feel more real. They’re showing life from before their impostor started posting. Many of these accounts popped up in late November, alongside the release of Google’s Nano Banana Pro image generation model. It’s crazy how many attractive twenty-somethings just happened to discover Instagram at that time. But, the “growing up” format was common for safe-for-work AI accounts, too. It’s also popular for real people posting nostalgic videos (more later on why you really should not do this).

But with hindsight, there were immoral creators taking advantage of this format and pushing it to its limit. On December 25th, I came across the first example of the format that troubled me so much, I had to take some time off to process it. What had I done to get this? How big was the problem?

How Riddance conducted this investigation

Allow me to break the fourth wall: This section is in-the-weeds of OSINT (Open-Source Intelligence) techniques. If you catch your eyes glazing over, skip ahead to the next section. But, for both transparency and further investigation, we decided to include it.

I found 11 offending videos on my main Instagram account between December 25th and 28th. After realizing the severity of the trend, we spun up a system to capture similar videos securely and responsibly. Our goal was to capture the scope of the problem, trace algorithmic ties between videos and accounts, and limit personal exposure as much as possible.

While a deep dive on our techniques is available in this post, here’s a summary how of we ran the investigation:

1. While using residential proxies with randomized IP addresses, including one outside of the USA, we created (3) Instagram accounts: The first on January 9th, the second on January 21st, and the third on January 23rd. Each account used different demographics and names.

2. We accessed Instagram on desktop browsers, though isolated Google Chrome profiles with Riddance’s in-house “Session Recorder” Chrome extension activated, which kept logs of our activity. After our first account tests between January 9th-15th, we started seeing videos with real minors. Understanding the significance,, we screen recorded the account creation and browsing processes for the second two accounts for more evidence.

3. After account creation, we “seeded” accounts by navigating to 10 offending videos, watching them all the way through, engaging with them by either liking or saving them, and following the offending accounts. We proceeded to browse and allow Instagram’s suggestion algorithms to present us with new content. We scrolled the Instagram home page and interacted with only those videos that met the criteria of “AI sexualized video including a minor’s likeness.” When encountering an offending video, we sometimes clicked on the account to verify that they consistently posted only AI-generated material.

4. On accounts (1) and (3), we saved offending videos and followed the accounts. On Account (2), we “liked” these videos and followed the accounts. As expected, “saving” videos encouraged more similar recommendations more quickly. “Liking” them yielded similar videos, but took more time.

5. Account (1) logged 40 minutes of browsing and interaction before stopping. Account (2) logged 41 minutes, and account (3) logged 22 minutes.

6. Browsing concluded when we hit “Rock Bottom” (described later) on all three accounts. After this, we revisited the saved videos and followed accounts, reported notable offending videos, and double-checked that we were only accounting for AI videos.

7. Finally, we combed through the most common and notable accounts to gather data and view counts on any qualifying “AI sexualized videos including a minor.” The 144 videos we collected were not all viewed during the browsing period, but were collected to assess the wider impact of the trend.

Some further, essential notes about our process:

1. Browsing sessions were kept short to limit exposure and personal impact, logging no more than 30 minutes at a time and spaced out among days.

2. All account and browsing was done by a human. We don’t use automation tools because human judgment is needed the entire time. We comply with Instagram’s Terms of Service, and we need to avoid any Instagram-side red flags raised by suspicious automation tools. As a result, browsing and collecting data was tedious.

3. We collected video URLs, account names, view count data, browsing session logs, screen recordings, and videos for reference. This data is only shared with professional collaborators, Meta, and the National Center for Missing & Exploited Children. We encountered no explicit CSAM, but have taken security and reporting seriously regardless.We reached out to Meta for comment and have not received a response yet.

The statistics

While a ton of AI-generated accounts start up every day, we tracked the most commonly encountered accounts and videos from our three tests. We anonymized twenty of them and ranked them by the view counts of offending videos.

In total, we qualified 20 Instagram accounts with 144 offending videos accumulating 130,625,138 views total.

Notes: As of February 10th, account “A.K.” was offline. Instagram has a variable rounding error depending on the view count. For example, a video with 12,315,239 views displays as “12.3 M”, while a video with 4,467 views displays the entire view count. We used the rounded numbers, because again: we did this all manually.

Of the 144 videos, 19 of them classified as “egregious sexual”, accounting for 10,353,200 views.

Many accounts had one or no “egregious sexual” videos, but these egregious videos seemed to help out accounts that dared go there. “H.D.” had seven, “L.A.” had two, and “S.O.” had two. These are three of the top five accounts by total views.

These numbers aren’t meant to demonstrate the entire scope of these videos across all of Instagram. We could kept finding more videos and accounts, but to what end? Furthermore, we only counted videos that were unambiguous. This means we only tallied videos that:

• Started with a girl who was clearly under age

• Mentioned their age was 17 or younger in the caption or a graphic

• Started with underage girl(s) only, or a girl with an adult man.

If the first frame had the underage girl with her mom or grandma, or if the girl was ambiguously teenaged, it didn’t count. If we had counted every video that included an underage person at all, the numbers would have at least doubled within these 20 accounts alone. We could have done that, but we determined those videos were less clearly violating Meta’s community guidelines as they’re currently written.

Hitting “Rock Bottom”

When we started this investigation, we hypothesized that there was an abundance of AI-generated accounts using the offending video format. The theory was: at some point, our feed would become saturated with AI-generated sexualized videos with the likenesses of underage girls. We would then report these videos to Meta, write a story about their reach, and move on to other investigations. We knew this would be unpleasant, but our capture tools and media expertise gives us an edge in finding AI videos.

We didn’t know “rock bottom” existed.

As I mentioned earlier, we used three test accounts for browsing and collecting data. We stopped testing whenever we hit “rock bottom” with each account. “Rock bottom” is the when the recommendation algorithm reliably and repeatedly suggests posts with real underage girls. Some of these accounts are run by girls themselves, but most are family accounts run by parents. None of their posts are sexual, nor is there evidence that they willingly or knowingly appear in pedophilic algorithms.

Once a feed’s balance becomes:

• 40-50% AI-generated sexual content, most of it with underage AI girls,

• 30-40% real sexual content,

• 10-20% real underage girls, out of context but surrounded by sexual content,

• 5-10% random videos, advertisements, apparently unrelated to the rest of the content,

we’ve hit “rock bottom.”

And that’s when we stopped the tests. The journey to rock bottom is already horrible; I don’t want to spend any time there.

We originally intended to use just one account to capture AI videos. But once we discovered “rock bottom”, the story changed. We decided to replicate the test with multiple accounts to see if getting to “rock bottom” was a common outcome that we could responsibly report. Indeed, we saw many of the same AI creators, and real girls, on all three of our test accounts.

We can apply a broad understanding of recommendation algorithms to understand why this happened. The algorithm does not “detect” that the content depicts young girls, nor does it explicitly “try” to show more. Instead, the algorithm correlates the viewing habits of the test accounts with other pedophilic viewing habits, and serves new videos accordingly. Most viewers don’t discriminate between real and AI videos like we were trying to, so the algorithm served us real videos anyway. We did not interact with the real videos.

Instagram’s Policies and Content Moderation

Meta has policies that should restrict the offending AI videos from Instagram. I’ve put polices that they appear to violate in bold. First of all:

DO NOT POST:
Child sexual exploitation

Content, activity, or interactions that threaten, depict, praise, support, provide instructions for, make statements of intent, admit participation in, or share links of the sexual exploitation of children (including real minors, toddlers, or babies, or non-real depictions with a human likeness, such as in art, AI-generated content, fictional characters, dolls, etc).

This seems straightforward: “AI generated” child likenesses are not allowed in sexual contexts. This is in accordance with laws banning AI-generated Child Sexual Exploitation Material (CSAM). The policy continues:

This includes but is not limited to:
[Sexual intercourse policies]

• Children with sexual elements, including but not limited to:

  • Presence of aroused adult

  • Presence of sex toys or use of any object for sexual stimulation, gratification, or sexual abuse

  • Sexualized costume

  • Staged environment (for example, on a bed) or professionally shot (quality/focus/angles)

  • [Other bullet points not relevant to this topic]

Maybe I’ve mistakenly applied an overly broad view on what these policies are for. If that’s the case: it’s time for some new policies.

Optimistic that maybe no one had thought to report these videos to Instagram yet, I reported 23 videos during testing. Only one was taken down. Six others were “Limited to viewers above 18 years old,” and the rest were left as-is. The one that was taken down was an “egregious sexual” video, but it was similar to the others I reported. And, to be clear, the “egregious sexual” videos may just add the “presence of sex toys or use of any object for sexual stimulation.” Every other video that I tallied violated the rest of the highlighted material.

What to do about it

Recommendations for you, the reader

Don’t post photos or videos of real kids on social media.

Some of the real posts we came across are from small accounts with less than 2,000 followers. Others were from large family accounts with over 500,000 followers. This includes nostalgic videos. A video from a mildly well-known real creator who a posted her childhood highlights showed up on accounts two and three.

The dark irony is that AI video companies like Kling have AI models that specialize in face or body control. You can submit a photo of a kid to make them do a “cute baby dance” or “expression challenge,” both viral dances. Seven of their twelve examples show kids or babies - it’s a focus of theirs. They’re using their technology to enhance kids’ likeness on social media, while ingesting these real kids photos into their training data for future AI models.

Recommendations for Meta:

Hopefully, Meta investigates offending accounts (which we provided to them) and interrogates the patterns that create this pipeline. While we acknowledge that moderating around 100 million photo and video posts per day is daunting, here are two concrete suggestions that we believe could mitigate harms in the future.

First, focus on moderating the most notable offending accounts through algorithmic capture. Disincentivize accounts who exploit the algorithm by “decapitating” the top offending accounts. As we’ll discuss in the next section, AI accounts are usually algorithmic wave-riders. Removing the most prominent accounts and videos not only disrupts the wave, but disincentivizes others from trying to ride it. But, you need white-hat, human auditors hopping on the wave for you, because automated tools lack the appropriate judgement.

Second, funnel more resources into third-party auditing and moderation. Great examples include Take It Down, a third-party collaboration between Meta and MCMEC, a platform “for people who have images or videos of themselves nude, partially nude, or in sexually explicit situations taken when they were under the age of 18 that they believe have been or will be shared online.” Meta works with other third-party vendors to weed out nuanced content, and the findings in our investigation should fall in that category.

We believe incentives for platforms and watchdog organizations are aligned on this issue, and given the obvious and extreme harms of CSAM, as well as the pipelines leading to it, we’re hopeful for rapid progress.

Gaps in AI and algorithmic accountability

This isn’t just an Instagram problem. We reflected on the systemic failures I encountered throughout this investigation. The problem is on track to worsen as AI media generation improves and more AI content works its way to social media.

The fundamental, high level problem is that there are no good methods for enforcing moral responsibility at points of contact between two or more autonomous systems at scale.

Companies making generative AI video models should do everything in their power to limit problematic content. But, a creator with bad intentions only needs to stitch together two AI videos that are potentially unproblematic in isolation to make content that is disturbing and downright harmful. If these videos were shot and edited by humans, many of them would be appropriately rejected, or not considered in the first place.

But the most sinister problem lies in the feedback loop between algorithmic engagement and automated, AI-generated content workflows.

There’s an explosion of automated AI accounts that operate with as little human input as possible. The topic and style of their videos is determined by the performance of other videos, either from the same creator or from channels the creator wishes to emulate. By indexing on well-performing videos, these creators automate prompt workflows to generate similar content, in hopes of riding the algorithmic wave.

The creators themselves have little interest in the actual content; they optimize only for engagement. If and when there is no human in the loop, the model-level guardrails are blind to the higher-level abuses they’re participating in when optimizing for views. There are no human editors, actors, or employees with the indispensable capacity to look outside of the scope of their specific role and step in and say “no, I will not be a part of this.”

Until AI and social media companies can effectively collaborate to prevent risks systematically, or regulators force them to do so, it’s imperative that each individual company over-indexes on safety and act with an abundance of caution. At this moment, it’s important to interrogate why AI-generated kids are appropriate or necessary at all.

Special thanks to Mason Broxham (editor and Riddance co-founder), Drew Harwell (editor), and Eli Broxham (illustration).

How to make AI spotting easy

Finding AI videos is hard, but finding AI accounts is easy. Anyone can do it with some basic knowledge about how AI media has advanced over the past two years.

I often joke about how many beautiful 24-year-olds discovered Instagram for the first time in November of 2025. Google’s Nano Banana Pro was released November 20th, and a rush of fake people registered. Let’s turn this into a practical way to find fake accounts. Put a pause on finding individual AI videos and AI videos; it’s more efficient to find the AI accounts, then work backwards.

In this piece we’ll plot out the timeline for recent advancements in AI generation and how new accounts and videos coincide with each change.

The Account Age Paradox

It’s obvious to us that an iPhone 17 Pro’s camera looks better than the camera from the iPhone 6S. 10 years of technological advancement separates them. But in 2015, a new iPhone 6S produced photos notably better than the HTC One M7 I had at the time. That HTC One took the earliest photos in my current photo library, and I look back on them fondly.

The same cannot be said of an AI creator in the year 2035, who is trying to prove their character is real. Can they look back at the old, Veo 3 AI generations from 10 years ago to establish proof of life? Of course not - it has the opposite effect, because Veo 3 looks awful and obviously AI to people in 2035. This is a huge difference between AI and real media, and one that AI creators are already aware of.

On Instagram, relevant AI accounts are usually just a few months old and pop up with AI media innovations. When I find a suspected AI character on TikTok, which unlike Instagram or YouTube does not make “account age” visible on the app, I immediately scroll down to their earliest posts. Did they leave the old, bad generations, or did they delete them and start over recently? Either are huge red flags and nearly impossible to overcome.

Putting it into action

I just found a new account on Instagram: an AI influencer named Olivia Arizpe. First of all, her bio says “Digital Creator” and “AI” in it, but let’s pretend the creator wasn’t so honest.

First, I’ll tap on the three dots at the top of the account page to pull up “About this account.” Here I see that the account was created in January of 2026. The first video is also from January of 2026. This video shows the AI avatar lip-syncing to a Tame Impala song. This is follows “motion control” trend I’ll talk about later. That’s all the information I need to confidently call this an AI account.

Or what about this account, a “News” page that claims to be from Tulsa, Oklahoma?

This page used to be called “Tulsa Area Breaking News”, but it’s just an AI slop page. They started in November, and their earliest available video is from the same month. This follows OpenAI’s release of Sora, and indeed there are Sora watermark scrubbing artifacts. Not to mention, their first video follows the format of the AI-generated EBT videos coming out at the time, and those were almost entirely generated with Sora.

None of this required good eyesight or any pixel peeping. And if any shiny, new AI slop comes out of the AI-generated Tulsa area and they really want to trick you, they’ll have to delete the old stuff.

We dive into how to find account transparency information in this article, by the way. It contains instructions for Instagram, Facebook, YouTube, TikTok, and X.

Granted, it’s unreasonable for you to remember that Sora came out in October, or that Motion Control AI generators were trending in January. Luckily, you don’t have to remember: I wrote it all down below for you to reference any time you need.

Subscribe now

The practical AI release timeline

When coming across an account we suspect is AI-generated, we’re looking for either:

1. A relatively new account that appeared and immediately implemented some recent technology

2. An account that shows signs of older or more affordable AI technologies in older posts

Let’s look at previous major advancements in AI media and how those popped up on the Internet. This will not only demonstrate how new technology changes trends on social media, but also what to look for when looking back at old posts. This will also give us some important cutoff dates. For example, if you see a dance video from 2023, you can be sure it wasn’t AI-generated.

Before 2025

2025 will probably go down as the most pivotal year in AI media history, and most of the AI-focused accounts today don’t use models from 2024 or earlier. But there are some notable exceptions you should be aware of.

Until Late 2023 - Deepfakes and AI avatars

In 2018, deepfake technology was creating memes and porn, but it was very much in the experimentation phase. Deepfakes aren’t AI videos — they use a different tech architecture — but they’re the first popular form of “AI video” in its broadest possible definition. The corporatized versions of deepfakes are usually called “Avatars”. These are made by companies like Synthesia, who were already around by 2020, and companies like Heygen joined the mix later.

Important takeaway: Face-centric “AI videos” were already possible before 2023, but their uses were pretty limited.

Late 2023 to Early 2024 - AI Image slop starts hitting social media

A lot of Facebook posts still have I would call “classic” AI slop. These are the images created by technology from late 2023. For example:

• OpenAI released DALL-E 3 in October of 2023

• Midjourney released V6 in December of 2023

While these are just two of many image generators of this era (and OpenAI is depreciating the DALL-E 3 API shortly), there are many accounts still stuck in this era because it’s cheap! These sorts of images became ubiquitous when OpenAI integrated DALL-E into ChatGPT, lowering the barrier of entry and opening the AI slop flood gates.

This is the era of Shrimp Jesus and “engagement slop” like old, disabled puppies asking for donations, or ragged puppies begging for likes. Today, we still see NFL coaches who desperately need help with medical bills. They don’t look like the coaches exactly, there’s a yellowish tint, and it’s giving “Polar Express.”

Important Takeaway: If you see a decent image from before mid-2023, its foundation is probably a real photo. Maybe it’s a heavily edited real photo, drawn, or 3D rendered by this era’s already incredibly powerful animation or gaming engines. But it wasn’t made by “AI”.

Early to mid 2024 - First (decent) AI Video models released publicly

Sora got a lot of early attention February of 2024, followed by models like Luma Dream Machine and Runway Gen-3 in June that year. AI video was still relatively limited, but boosted by many new scaling and training advancements from 2023. Given its limitations, memes and AI video slop was their primary social media use.

Important Takeaway: If you see a decent-quality video posted before mid-2024, unless it was professionally modified or rendered, it’s a real video.. There aren’t any good “AI videos” before this point.

2025

March - ChatGPT 4o Images

I still remember sitting on the couch with my wife on March 26th, 2025 - the day after OpenAI released ChatGPT’s 4o image generation. The internet was lit up with Studio Ghibli-ified images. Feeling guilt about the inherent ethical quandaries of that style, I instead made a photo of our recent vacation in the Family Guy art style. Our surprise and fear from that day seems pretty quaint in hindsight, given what else 2025 was about to bring.

AI-generated profile pictures from this era are still common. The yellow-tinted, glossy, evenly-lit photos from this generation are ubiquitous. Since ChatGPT is the most popular large language model, its built-in image generation is also really popular.

May - Google Veo 3

Veo 3 was the first AI video model with meaningful sound, and it was also a jump forward in video quality. It generated videos with a cinematic look and feel, though it definitely wasn’t up to cinematic standards. As a result, though it was envisioned as a tool for creatives, it instead flooded the internet with AI videos.

A Veo 3 Review was my first ever YouTube video. Today, Veo 3 makes a platonic “AI video” that’s relatively easy to spot after you’re familiar with it. The most distinctive characteristics include smooth and even lighting, temporal inconsistencies and background issues, and the character’s robotic and melodramatic voices.

Immediately in May, a ton of AI slop pages popped up on social media. Since Veo 3 could only do widescreen 16:9 videos at first, vertical-native platforms like TikTok suddenly had a ton of new accounts posting AI videos with black bars (also known as letterboxes) on the top and bottom of every video. ASMR Fruit cutting videos and AI man-on-the-street interviews were trend. But also, people’s first “I got tricked” moments came with Veo 3’s bunnies on trampolines.

Important Takeaway: If you find a realistic video with matching sound or dialogue posted before May of 2025, unless it was generated with a game or animation engine, it’s a real video.

Mid-2025 - Other video models

Around the same time as Veo 3 were Runway Gen 4 (April) and Kling 2.1 (May). These were similar in video quality to Veo 3, but neither had synchronized sound. They traded this for vertically-native videos and different video styles. Along with Midjourney releasing its first video model in June, Alibaba’s open-sourcing of Wan 2.2 in July, and many more advancements from companies like LTX and Minimax, a plethora of good video-generation options came online in mid-2025.

With these vertical-native models, we got AI-generated landscapes, fake natural phenomena, and AI-generated cartoons for kids that were NOT kid-friendly. But this is also when AI slop started looking realistic enough to fool a ton of people in vertical feeds. By August this is almost all I covered.

August - Nano Banana

Google’s Nano Banana, the informal name for Gemini 2.5 Flash Image, was a big jump in photo quality. Before Nano Banana, mainstream AI photo still had a glossy look. After Nano Banana, photorealistic images were very accessible.

Important Takeaway: A lot of photo-only AI accounts got a huge boost or started in August of 2025.

October 2025 - Sora 2

The next jump in AI video came from OpenAI, who took a year and a half after the original Sora to release the Sora 2 video model. Alongside it was the release of the Sora app, a TikTok-style vertical video app with only AI videos.

The Sora 2 model had a few key innovations:

• It felt less uncanny than Veo 3 for many viewers. Eyes, mouth, and skin detail were more realistic.

• It had better physics than any other model at the time.

• It was funny. Lazy prompts were spiced up by a language model in the background. This meant inexperienced AI prompters could make viral videos.

And yet, it was a very noisy model with a heavy “AI accent.”

Sora’s release coincided with a huge increase in the number of “AI slop” accounts because it was free. Until this point, good AI video generation was expensive, but the Sora app let users generate a lot of videos in the free app. These videos could be downloaded with a watermark (that was easily scrubbed), then reposted to the other, much more popular social media sites. The Sora 2 API released just 2 weeks after, providing videos without watermarks, and giving more people access to the powerful Sora 2 Pro model. To this day, a ton of AI videos come from Sora 2 because it has a good price-to-quality ratio.

Important Takeaway: Many AI video slop accounts have October 2025 birthdays or changed their usernames at this time.

November 2025 - Nano Banana Pro

Google’s jump to Nano Banana Pro (Gemini 3.0 Pro Image) was surprising. Just three months after the original, it brought big improvements to realism and quality, as well as improved spacial reasoning and text rendering. This is the point where AI photos became mostly undetectable on first glance, though they can be spotted when looking closely.

Along with Nano Banana Pro came the release of SynthID, which can be accessed through Google’s Gemini large language model. It’s an invisible watermarking system that embeds and detects a watermark hidden inside the pixels of a photo.

Creators reacted accordingly. Misinformation like the infamous “Bubba Trump“ photo and fake celebrity paparazzi photos spread wildly. More relevant to our everyday social media use, AI generated influencer accounts proliferated in late November. The Nano Banana Pro release also corresponded with TikTok’s unfortunately-timed new emphasis on image carousel posts. And, since many AI video generators have a photo-to-video mode, these AI images are the starting frames for many AI videos. Nano Banana Pro is still a leader in this field, and while other generators have caught up, this was an important demarcation point.

And that wasn’t all that happened in November...

November and December 2025 - Motion Control AI improvements

Coinciding with Nano Banana Pro’s release were improvements in Motion Control models, most notably Wan 2.2 Animate. This prompting innovation lets creators take a “control” video — often a video stolen from a real creator’s social media — and “replace” them with a new character, real or not. This method saw further improvement through Kling 2.6 Motion control.

These models unlock very realistic motion and physics for AI characters. Combining Nano Banana Pro (or similar photo models) with a motion control video model lets creators generate realistic, consistent characters across posts. While not perfect (there are still plenty of AI giveaways), AI influencer creators now had a ton of tools at their disposal.

Important Takeaway: Accounts that feature human avatars and started or rebranded in November or December of 2025 are a huge red flag. A lot of AI slop accounts moved into the more profitable AI influencer business.

Early 2026 trends

Releases from Kling and Bytedance show further improvements in AI video quality, which we’re still analyzing as of this post. I expect some new accounts in February that play off their strengths, but those are yet to be determined. Right now it’s a bunch of demos of stolen intellectual property and celebrity deepfakes, as can be expected with a new model release.

Moving Forward

People regularly ask me what might happen if AI video becomes “perfect” or “undetectable.” But to my eyes, real videos aren’t “perfect”. There are always artifacts of the process that made them.

I remember watching Superbowl XLV 15 years ago, on an imposing 75-inch high-definition TV. It was incredible and perfect at the time. But looking back at it now, it looks a bit dated, even without today’s de-interlacing conversion artifacts. In 5 years, perhaps the oversaturated HDR look of today’s iPhone phones will be an out-of-fashion artifact of our current era.

The highest-end, most impressive AI video generations today still don’t look "real” to me, but they do look “really good”, which is real enough for most people. The difference is marginal, but with hindsight it may become obvious. Those of us who make real media will have to adapt and differentiate ourselves from AI advancements, figuring out what our advantages are. It’s always going to be changing, so stay tuned here for updates on the latest releases and countermeasures.

Even making critical, expert-level judgements or uncovering large and imposing AI-generated networks, we want readers leave with tools they can use in their own lives, using many of same methods that we use. This isn’t the primary goal of most news outlets, and it’s pretty hard to do.

This article will explain exactly what we do and how we do it. We’ll focus primarily on videos. It’s tedious to say “video, photo, audio, and text” all the time. But similar techniques may exist in photos, audio, and text, even if we don’t call them out specifically.

Media Analysis Methods

We’re proud of our in-depth knowledge of video and audio technology. It’s an edge we have from years of practice. Regular people (that’s not an insult - be glad you can watch a movie in peace) can learn the basics, then go more in-depth when they want to.

From understanding how real cameras and microphones work, to keeping up with the state-of-the-art in editing techniques and AI generation, we know what to look and listen for to distinguish real from fake media. And yet, media never exists in a vacuum. Its chain of custody will tell you where it came from, why someone shared it, and what parts of the process humans were directly responsible for. That’s why the video and the context are considered simultaneously.

Video analysis

During video analysis, we search for evidence of real video, edits, AI-generation, or all of them at once.

Real videos originate from a physical space, captured through a lens and sensor, then are recorded onto a film or digital format. Film, though less common today, is special because it shares a direct, physical relationship to the light that exposed it. Digital cameras approximate that physical interaction by recording videos into bits through sensors, and they can do it incredibly well. Real sounds originate from a physical space, too, and they can be captured with microphones and onto a recording devices. These physical processes and the conversions have “fingerprints.” People have always tried to get rid of these fingerprints because they can be distracting, but that’s almost impossible to do.

In real videos, some examples of “fingerprints” are lens flares, sensor noise, focus pulling, and sensor stabilization. Less obvious are the parallax and light interactions that exist in the physical space, and how they relate to a camera’s physical vantage point. This was taken for granted for a long time, but AI generated videos don’t fundamentally have these characteristics - they just “learn” what they look like.

Modern AI video models use algorithms to create pixels. These pixels aren’t formed by a complex relationship between physical capture, encoding, and editing, but by a full-stack synthesis of the video itself. This is typically done with a process called diffusion. When AI-generated videos create strange perspectives, breaks in physics, or just obviously mess up what real life looks like, these aren’t just “mistakes”. They’re “fingerprints” of the algorithmic process.

Editing analysis

Videos have been edited for as long as they have existed. Editing isn’t usually deceptive; it’s essential in most video formats. Whether it’s a simple cut between scenes, a lightweight text caption added on top, a filter applied to smooth out the skin, or complex graphics compositing, we separate out original media from the editing techniques.

There are infinite combinations of techniques that make this much more complex than just saying “That’s real! Case Closed!” Animated films and synthesized instruments have been an intended “gotcha” point of many “this is real” debates in the age of AI-generated media, but we track human touch and transparency more than we care about the “authenticity” of specific mediums.

So, by analyzing the source material and editing techniques, we can distinguish between an AI-generated influencer and a real person using a touch-up filter. An AI influencer is objectively not real because they have no physical presence. The real person, captured by a lens and camera, then edited with a skin filter, is still a real person. Are they being their fullest, realest self? That’s not important in our judgement of if they’re real, though it may be a useful data point in other ways.

Metadata & AI detectors

We’re usually viewing media through social media platforms, and they don’t usually make metadata available. Metadata is hidden data in a file. It can contain very useful information, often down to the time of day and exact location where the video was shot. We jump at opportunities to get original files. Otherwise, we can download videos directly from the platforms to get more generic metadata like a video’s aspect ratio, frame rate, video codec, and more.

We run suspected AI photos through Google’s SynthID using Google Gemini. Otherwise, we rarely use AI detectors for more than a small data point. We don’t recommend AI detectors for most social media use cases, because there are so many edits and processes that are common in social media, and many AI detectors can’t reliably detect those. If we’re feeling confident about a specific AI method used and are familiar with an AI detector that is proven to detect that method, we’re more likely to use it. If we suspect that the AI method used is state-of-the-art we won’t use AI detectors at all.

Source identification

When determining authenticity, finding the source is just as important as analyzing the video itself. We look at what the source’s patterns and techniques are. This necessarily means that we authenticate many other videos, photos, and even text put out by the source, to get the big picture about what they do and why. We look for signs of where humans or machines are involved in the process.

Page analysis

Always take what the social media platforms give you! For example, on the front end of Instagram, you can view an account’s age, number of previous handles, and account location (if the user allows it). Most AI-focused accounts were created within the past year, or have a previous handle as they re-branded for their new AI brand. On the other hand, a real creator who has been posting since 2012 is pretty safe. Of course, real creators experiment with AI generation, and real people are starting social media pages every day, too.

Platforms have different transparency levels. Our full guide is elsewhere on Riddance, but in short:

• Facebook, Instagram, and YouTube have easy places to find account age and location

• TikTok doesn’t publicly show account age and location, but APIs make this information available.

• X is always changing

It’s often as simple as scrolling back to their first public post, looking for location tags, and doing some basic browsing.

Linguistic and behavioral analysis

A huge amount of information is stored in the text descriptions and copy. Do the language, idioms, and vocabulary align with their claimed identity? Do strings of text show up on other automated channels, or are they unique? Does the text show signs of AI-generation? These help determine what the entire content workflow looks like, as automated channels usually try to automate as much as possible.

Some of us were doing source analysis with just Tweets, long before AI videos were a concern. Relative to looking for bot farms on Twitter, videos or photos have so much information to work with. How often are they posting? Do their posts align with other similar posts from automated accounts? Is their posting time aligning with their expected time zone? There are so many other places to dig in.

Website information

If they have a website, we’ll do a DNS look up to see who owns the site, when it was registered, and what other information we can gather there. A website itself has a ton of information. A simple “link in bio” can link you to other pages that a creator forgot to monitor.

AI video benchmarking

Keeping an eye on what’s happening with AI by following others is not usually enough. We run tests frequently.

If there’s a video in question that we think could be AI-generated, we’ll try to replicate it. We’ll approximate the model(s) that could have been used and try to get a similar-looking video. Since AI models are random to varying degrees, you’ll never get an exact match. But seeing similar types of artifacts, subjects, and styles is really helpful.

We regularly benchmark new models as they’re released. We use API services like fal.ai to access a ton of models at once and only pay for what we need. We’re working on ways to give more people access to this testing. Stay tuned.

Subscribe now

Trend Analysis Techniques

We’re often collecting data and analyzing trends to judge the impact of AI media. This means browsing social media platforms to collect information. How we collect that information is important.

Everything is manual

While it would be really efficient and make our jobs way easier, we avoid automation tools for browsing social media accounts. There are three reasons for this:

1. We want human judgment at every step. We’re assessing automation and AI technologies that can’t be reliably detected by machines.

2. Automation changes algorithms. For example, TikTok’s recommendation algorithm treats you differently if you scroll through a video exactly at its conclusion, versus scrolling when you just lose interest. In rare cases, we need machine-like precision (for example, if we don’t want view duration to sway recommendations), but we’ll still try to find analog methods rather than using a bot.

3. Automation tools are against the terms of service for most social media platforms. Humans need to click the buttons.

Riddance investigation tools

We have some proprietary software. First is a custom browser extension that helps track sessions, video metrics, and tie video data to account-level data. It helps us store data in a way that is persistent, auditable, and shareable. We store videos across different platforms in a database, tying together accounts owned by the same people and keeping a “navigational graph” of a browsing session.

One idea that we think is important is information symmetry. Having a clear picture of the kinds of information a platform gleans from you is a powerful mechanism by which users can take back their autonomy and agency from platforms. The browser extension helps us mirror what the platforms are collecting about us while browsing.

The navigational graph that the session recorder tool spins out is designed to let our analysts retrace their footsteps throughout an investigation. If we find a video title on YouTube and search for that same title on TikTok and it surfaces another video of interest, we can link these two videos. Sometimes you’ll see this graph in our reports, because it’s a great visualization for networks of videos. We preserve this graphical structure in a database to extract for analysis later and to collaborate on investigations.

Recording saving media

In addition to saving logs with the browser extension, we may screen record sessions either with a software or hardware recorders. This helps us preserve every bit of information possible. We save video URLS for investigations, then download video files for video editing and archival. This is usually through a command-line interface like yt-dlp, which also allows us to route downloads through proxy services. This is also how we ingest most of the videos needed for video edits. We’re really sensitive about how much media we save and we make sure to credit the media we use.

Creating fresh accounts for browsing

Creating fresh social media accounts is essential. They allow us to browse platforms free of biases from our existing social media accounts and control different variables. For example, creating a new YouTube account from the same IP address as your other accounts may influence the initial YouTube recommendations. This type of tracking goes much deeper than people expect.

There are a few goals we may have when creating new accounts:

Find similar media through targeted searches

“Does this exist somewhere else?”

Whether it’s finding reposts of a single video, finding multiple accounts associated with one actor, or understanding how groups of uncoordinated actors are riding an “algorithmic wave,” we use techniques like quote searching a string or following hashtags. Sometimes it’s overkill to do this on a fresh account, but existing account data can sway your results.

Find similar media through by recommendation algorithms

“How much more of this exists?”

What are people seeing after they engage with a video or photo, but don’t seek it out explicitly? We want to see what recommendation algorithms will show them. We do this by “seeding” a fresh account in an isolated environment. After creating the account, we engage with a specific type of content, hoping an algorithm picks up on what we’re engaging with and “reacts” to it.

Understand what others are seeing

What are people seeing in general? For example, on a fresh account with no follows, what are the first 50 posts you get? How does that change when you provide certain collection of expressed interests and follow specific accounts?

Investigations are full these types of searches, which also yield a lot of AI-generated media to teach with.

Creating the accounts

Web traffic

We send web traffic through residential proxies, which is a fancy way of saying “we make it look like we’re using the internet at someone else’s house.” There are tons of proxy services out there, and we don’t recommend one over another, but having one that hasn’t yet been overused by scraping services will make everything go smoothly later. Occasionally we’ll pick proxy locations outside of the US, and we’ll cycle through different locations in general.

While using a popular VPN might be tempting, social platforms recognize most of their addresses and either block them or make it a big pain to use them. So, while using a VPN and a Proxy together provides good backup in case a proxy should fail, just the proxy is necessary. We usually route proxy traffic through a Chrome extension on Desktop and route all of the phone’s traffic when using a phone.

Unfortunately, being more specific presents some security and privacy issues. We also don’t want to give cybersecurity recommendations - that’s not our field.

Devices and accounts

We keep burner phone numbers, create new email accounts, and use new names and account information every time. While using individual Google Chrome profiles per session and tying proxy sessions to each profile is usually enough, occasionally we’ll use entirely new devices or virtual machines. Our methods change for the situation because we’re never doing anything illegal, we just want fresh accounts with no ties between them.

Everything evolves

Our methods are always changing with the content, platforms, and tools available. We hope this article gave you a good idea of where to start doing your own investigating. Check back for updates occasionally. We’ll create a change long at the bottom when appropriate.

How we got here

AI-generated media moves too quickly for everyone. The social media companies don’t know what to do. No one has reliable detection solutions. Laws haven’t kept up. It feels like those responsible for AI media’s advancements are looking for others to deal with the damage left in the wake.

I started the ShowtoolsAI social channels in June of 2025 to talk about AI video. My objective was unclear at the time, but I believed AI-generated media could be a “tool” for “shows” in the best-case scenario, so long as it was used ethically. Funny enough, I registered the showtools.ai domain one day before Google’s Veo 3 came out. Oh, to be young and naïve…

I’ve been talking about how to spot AI videos since then, using the skills I developed as a live-streaming technical producer, camera operator, and sound engineer to discern real from AI. After building encoder tests and demonstrating camera shading methods, then producing hundreds of multi-cam live streams, I had (unfortunately) learned everything that could possibly go wrong with real videos. AI video looked like completely different — it might as well have been a video game.

As the gap between real and AI videos (and photos, music, text…) narrowed, our methods had to adapt.

Building a team and method

Mason Broxham texted me on July 3rd 2025, the day he left Microsoft. A friend since college, I knew him as an academic who studied linguistics and philosophy, who then transitioned into tech as a threat analyst specializing in AI detection. The need for AI literacy and detection was shifting. Once just an enterprise and cybersecurity concern, it was now essential for regular people to understand what was happening and what to do about it.

After a couple months of discussion, we started building. Mason got to work on an epistemic framework for tracking AI on social media, which turned into a database that stored AI videos and software to analyze and track them. Our goal was to capture the “fingerprints” of real and automated accounts, which turned into a full research methodology. Behind the scenes of my short video production schedule, we were building a system for long-term detection and tracking. We still are; the tools aren’t publicly available yet.

So, creating a place where we could share research, investigations, and educational material was the logical next step.

Why we’re launching Riddance

It’s weird to go from video to text while most journalists are doing the opposite, especially when your subject matter is other videos. But there are a ton of reasons to focus on writing:

• The AI world moves really fast. Short-form videos were my answer for a long time, and they will still be an essential part of our reporting. But when you need to get important information out quickly, writing is much faster.

• The short-form video format is incredible for discovery, but it's really bad for nuanced discussion about media literacy.

• Long-form videos have an expectation of being durable or “evergreen”. My “How to spot AI videos in 2025” is already outdated 6 months later and still finds new audiences. Leaving it up for much longer will start actively doing harm (it may be down when you see this).

• The video recommendation algorithms are a huge part of the AI problem and might not be long for an AI-infested world. Who wants to see new things if they’re probably fake? That’s a hot take for another time.

• Video platforms probably aren’t going to like what we have to say about them, nor will our reporting always be appropriate for video.

But most importantly, writing and reading are awesome! I get messages all the time from teachers who want written material, or people who wish their parents could read our media literacy tips rather than watching them in quick, short-form videos. And when you’re conscious about the problems with video feeds, it feels good to just write and not contribute to that noise.

Our goal is to grow a full team so we can do more investigations and educational work. There’s no shortage of topics to cover, and a ton of demand.

Why you should subscribe

We will provide:

• News about advancements in AI media

• Analysis and investigations on how people are using AI media

• Toolkits for media literacy tailored for the AI age

That’s the baseline. We’ll make as many posts freely available as possible an appreciate your subscription.

If you can afford to support us, your paid subscription gets:

• Occasional paid-only posts

• Access to the chat and community

• Full access to the archive

We’ll keep posting videos. Maybe we’ll experiment with some live streaming or podcasting. We’ll keep evolving and will appreciate your company along the way.