Why companies lie about detecting AI writing

We know that AI cheating detectors don’t work, but what should educators do next?

AI detection doesn’t work, and never will

AI detection doesn’t work, and it never will. That’s the harsh, ugly truth. Anyone who says otherwise is misinformed or trying to sell you a product that doesn’t work. More on that later.

The evidence for this truth is easy to find and is blatantly clear to anyone who has looked into the issue. “AI writing detectors” have high false positive rates that are not transparently shown, and disproportionately flag writing from non-native speakers as AI-generated — which raises serious ethical concerns.

Perhaps the best evidence for this truth is that OpenAI, the creator of ChatGPT and the industry leader in AI models, discontinued their own AI detection tool with the notice:
“As of July 20, 2023, the AI classifier is no longer available due to its low rate of accuracy.”

If the company that has built the most powerful foundational AI models can’t detect the output that its own AI produces, what chance does a company working without that expertise or proprietary information have?

Two of the biggest “AI writing detectors” have both been proven to be inaccurate. ZeroGPT was shown to have a false positive rate of 20%, and TurnItIn walked back claims of accuracy after teacher backlash and has not published full accuracy results or allowed independent researchers to evaluate their software.

A September 2023 study from the International Journal for Education Integrity produced bar graphs showing the diagnostic accuracy of AI detectors given writing written by ChatGPT-4 as having a wide variance in detection ability.

The fundamental, unsolvable engineering challenge here is that the creators of AI models (OpenAI, Anthropic, etc.) can’t watermark the output of models with detectable patterns, and third parties are not able to reliably detect signatures of AI writing. As AI models continue to improve, the writing they produce will reflect creativity, variety, and structure indistinguishable from human writing. This means that the accuracy of current detection methods will rapidly get worse, not better.

Why do companies lie about AI writing detection?

The job of any company is to make a product that the market wants — to create a supply that addresses demand. The most innovative companies create products to meet emerging demands. AI writing detection is a massive emerging demand due to the massive emerging problem of students using AI to write.

Between schools and universities, there are millions of potential customers for AI writing detectors. The promise of a new multi-billion dollar market is incredibly tempting, especially if your target customers are educators who have an immediate need and lack the bandwidth to evaluate rapidly changing technology that is barely a year old.

Turnitin, for example, has spent over 25 years addressing plagiarism via similarity detection which is powered by internet sources and a massive trove of private submission data from over 30 million students. Seemingly overnight, due to generative AI, similarity detection became useless for preventing cheating. Instead of plagiarizing, students can generate entirely novel untraceable writing using ChatGPT in seconds.

Ultimately, we all want to believe that AI writing detection is possible because we’re afraid of change. We know that if AI writing can’t be detected, the future of teaching writing is in jeopardy. We’re too scared to rip off the band-aid, which makes us susceptible to deception from those who stand to profit by misinforming.

What should teachers do?

The only viable option for teachers is to embrace this change, as disruptive as it may be.

First, it’s important to recognize the reality that we’re in. Students will use AI to write, and there’s no way to detect if they do.
While this is concerning in terms of ensuring academic honesty, it’s important to remember that cheating on writing has always been a problem. The popularity of contract cheating through essay mills means that by 2017, over 31 million students had paid someone to write their paper for them.

What’s changed in the last year, then, is that cheating has now become more democratized because students can now cheat on essays for free and instantaneously thanks to generative AI.

Banning AI may be just as tempting as trying to detect AI writing, but it’s also not a realistic solution — especially when AI writing functionality is becoming embedded into everyday tools like Microsoft Word and Google Docs.

It’s important to understand that students use AI for writing because these tools are inherently valuable. For students, AI writing:

Helps them get a better grade
Helps them improve their writing

While #1 is a concern for fair and accurate assessment, there’s potential for real educational value in #2. If students are using AI to get more immediate and tailored feedback, we should dig in here to understand how to best use AI to systematically improve student writing skills.

The future of learning to write

Learning to write will always be important. Generative AI is not to writing what the calculator was to math. I love Dan Willingham’s rebuttal of the analogy where he explains how, unlike with math, the content and process in writing will always impact the outcome:

So, how do educators continue to teach and hone the skill of writing given the advent of generative AI?
In the long run, how we assess writing has to change to become more adaptive to every student, and conversational assignments seem like a step in the right direction — especially as a replacement for standard written responses.

For essay writing, humanities teachers have been using AI to provide students with much more immediate and tailored feedback on their writing, thereby removing a key bottleneck in the learning process.
By having open conversations with students about their use of AI in writing, teachers can set guidelines around appropriate use and start to figure out what the future of learning to write will look like for their students.

And, here at Flint, we’re committed to never building or selling AI writing detectors.