San Francisco Frontiers - AI-generated videos wield Down syndrome to make sales

The crush of posts hawking resin lamps, crochet handbags and clay bowls fits into a wider pattern of AI-generated clips that seek to score sales by stirring empathy for people who have disabilities or are otherwise perceived as disadvantaged, such as Black people or the elderly.

Advocates and researchers say this practice perpetuates negative stereotypes and harms the lives and business of real people.

Across TikTok, YouTube and Instagram, the AI characters claim they are being bullied for selling crafts against a backdrop of insults using disparaging language about people with disabilities.

Nathan Rowe, program director at Down Syndrome International, told AFP the videos play into the stereotype that people with the genetic condition caused by an extra chromosome need to be pitied.

"They're preying on people who have maybe a bit sympathetic, slightly paternalistic view of Down syndrome," he said.

The accounts link to suspicious online storefronts, including one touting multiple "five-star reviews" which all feature the same filler text.

AFP found some of the pottery products first appeared in content from real creators, before being appropriated. One crochet bag design may not be handmade, appearing for sale on Shein, a popular site for dropshippers, who are online retailers operating storefronts without keeping physical inventory.

The videos "crowd-out" posts from actual entrepreneurs with Down syndrome, potentially siphoning away business, Rowe said.

"There's lots of really talented people with Down syndrome out there who are making things, but it kind of reinforces the narrative that people with Down syndrome can't and it must be AI," he said.

- Ongoing trend -

This is not the first trend of its type, appropriating disadvantaged people for attention or profit online.

Down Syndrome International complained to Meta about sexualized deep-fakes of people with Down syndrome and this resulted in the removal of many videos. But Rowe said social media companies should be more proactive in preventing this kind of content from spreading.

AFP reached out to Meta about the new product-promoting trend but did not receive a response. According to TikTok’s community guidelines, the platform bans accounts engaging in deceptive or manipulative activity or discrimination. YouTube has similar policies about misleading spam.

While many of the videos AFP examined are no longer live, other accounts are still sharing the AI-generated clips redirecting users to products.

"The people behind the scenes are probably motivated by profit and have no regard for the damage they do in the process," Rowe said.

- Simple to generate -

Jeremy Carassco, co-founder of the AI research firm Riddance, said the number of accounts pushing this type of content indicates that the videos are working as a profit-making scheme.

"There's a lot of system-wide failures that are compounding to make this worse," he said, pointing out that the videos are exceedingly easy to create and difficult to keep track of.

"It's why these have exploded to the degree that they have."

He said there were countless videos featuring the AI-generated figures with Down syndrome and said the same accounts had been trying to sell identical products using elderly synthetic characters.

AFP previously investigated a multilingual trend of videos stealing seniors’ identities to bait users into sympathetic purchases of slippers and dog collars.

"It feels like we're hitting kind of the bottom of what is permissible, and if they keep going further, I think something's going to happen," Carassco said.

A.Evans--SFF