San Francisco Frontiers - AI-generated Pope sermons flood YouTube, TikTok

An AFP investigation identified dozens of YouTube and TikTok pages that have been churning out AI-generated messages delivered in the pope's voice or otherwise attributed to him since he took charge of the Catholic Church last month.

The hundreds of fabricated sermons and speeches, in English and Spanish, underscore how easily hoaxes created using artificial intelligence can elude detection and dupe viewers.

"There's natural interest in what the new pope has to say, and people don't yet know his stance and style," said University of Washington professor emeritus Oren Etzioni, founder of TrueMedia.org, a nonprofit focused on fighting deepfakes.

"A perfect opportunity to sow mischief with AI-generated misinformation."

After AFP presented YouTube with 26 channels posting predominantly AI-generated pope content, the platform terminated 16 of them for violating its policies against spam, deceptive practices and scams, and another for violating YouTube's terms of service.

"We terminated several channels flagged to us by AFP for violating our Spam policies and Terms of Service," spokesperson Jack Malon said.

The company also booted an additional six pages from its partner program allowing creators to monetize their content.

TikTok similarly removed 11 accounts that AFP pointed out -- with over 1.3 million combined followers -- citing the platform's policies against impersonation, harmful misinformation and misleading AI-generated content of public figures.

- 'Chaotic uses' -

With names such as "Pope Leo XIV Vision," the social media pages portrayed the pontiff supposedly offering a flurry of warnings and lessons he never preached.

But disclaimers annotating their use of AI were often hard to find -- and sometimes nonexistent.

On YouTube, a label demarcating "altered or synthetic content" is required for material that makes someone appear to say something they did not. But such disclosures only show up toward the bottom of each video's click-to-open description.

A YouTube spokesperson said the company has since applied a more prominent label to some videos on the channels flagged by AFP that were not found to have violated the platform's guidelines.

TikTok also requires creators to label posts sharing realistic AI-generated content, though several pope-centric videos went unmarked.

A TikTok spokesperson said the company proactively removes policy-violating content and uses verified badges to signal authentic accounts.

Brian Patrick Green, director of technology ethics at Santa Clara University, said the moderation difficulties are the result of rapid AI developments inspiring "chaotic uses of the technology."

Many clips on the YouTube channels AFP identified amassed tens of thousands of views before being deactivated.

On TikTok, one Spanish-language video received 9.6 million views while claiming to show Leo preaching about the value of supportive women. Another, which carried an AI label but still fooled viewers, was watched some 32.9 million times.

No video on the pope's official Instagram page has more than 6 million views.

Experts say even seemingly harmless fakes can be problematic especially if used to farm engagement for accounts that might later sell their audiences or pivot to other misinformation.

The AI-generated sermons not only "corrode the pope’s moral authority" and "make whatever he actually says less believable," Green said, but could be harnessed "to build up trust around your channel before having the pope say something outrageous or politically expedient."

The pope himself has also warned about the risks of AI, while Vatican News called out a deepfake that purported to show Leo praising Burkina Faso leader Ibrahim Traore, who seized power in a 2022 coup.

AFP also debunked clips depicting the pope, who holds American and Peruvian citizenships, criticizing US Vice President JD Vance and Peru’s President Dina Boluarte.

"There's a real crisis here," Green said. "We're going to have to figure out some way to know whether things are real or fake."

L.Nguyen--SFF