San Francisco Frontiers - Spotting AI writing: how reliable are the detectors?

An X account with a handful of followers sparked the controversy by claiming that the AI-detecting program Pangram had concluded the tale of a Haitian migrant travelling to Canada had been largely written by a text-generator such as Claude or ChatGPT.

Orelien, 38, denies the allegations and has said he will prove that he wrote the book, which has been sold to publishers in more than 20 countries.

"I didn't write this novel with my head, but with my guts and my heart," he told AFP recently.

AFP ran excerpts of the book through various AI detection tools and came up with a variety of results -- from it being "very likely" to have been written by a human to "100 percent" AI generated.

- How do AI detectors work? -

Dozens of tools are now available online to detect AI generated text -- many of them designed for use in education or publishing.

They generally have large troves of AI-generated text alongside content written by humans -- and compare the two.

Their performance has improved significantly in recent years.

In a paper published in October 2025 called "Artificial Writing and Automated Detection", researchers at the University of Chicago found Pangram "achieving near-zero error rates" on medium-to-long extracts.

- What are the clues? -

Thierry Poibeau, a researcher at France's national CNRS research institute and specialist in natural language processing, said the programs look for "linguistic tics".

"The classic example is triadic phrasing (using three clauses), or em dashes," Poibeau explained.

These telltale signs used to be common in generated texts because the tools had been trained on American scientific writing or American novels, he said.

Pangram claims it is able to "detect AI text by analysing and understanding subtle cues in the writing".

Some words or phrases show up regularly, while monotonous sentence structures are also a warning sign.

- Are the tools reliable? -

Many tools publish percentage figures about their accuracy, but the figures are impossible to verify and do not take into account the risk of a false positive.

Pangram CEO and co-founder Max Spero stressed to the American outlet The Atlantic that the tool should not be the ultimate arbiter, but rather a starting point for a more in-depth investigation.

According to Thierry Poibeau, these detection systems also run up against a "moving target problem" because AI programs are "constantly evolving".

The expert also points to a potential problem when it comes to detecting the use of AI in novels.

"The tool may pick up something particular in the writing, something repetitive, but that might be exactly what constitutes a writer's style," he said.

Orelien claimed this week his style was based on Haitian and Caribbean traditions, telling Liberation newspaper the software was "used to the French literary tradition and classical authors".

- What do the experts say? -

Benoit Raphael, an entrepreneur and writer specialised in AI, published his own research using Pangram to test Orelien's work -- and the results were also positive for AI.

"The fact that it rated at around 100 percent AI, or even 90 percent, is a pretty telling sign," he told Nouvel Obs magazine.

And although the book was widely praised by critics and was flying off shelves in France before the scandal, some doubts had been raised.

Lea Bory on her podcast Torchon was possibly the first to say publicly earlier this month that she thought it had been written by IA because of the repetitive nature of the sentences and the over-use of analogies that are hallmarks of IA.

Writing in newspaper Canard Enchaine, Fabrice Colin also criticised the formulaic sentence structure and said he "vainly looked for a unique voice" in the prose.

E.Jones--SFF