Using AI in submissions to check they have met journal criteria

Software Support
1

Has anyone experience or advice on how to set up a pre-submission review step. This would be for authors to upload their paper to the journal’s AI assistant which gives them feedback on how well or in which ways they have or have not met the criteria for articles. Referencing and writing styles, on topic, that sort of thing.

We have a checklist for submissions and good accessible contributor advice but it’s often not used. A stage like this would save reviewers and editors proffering the same advice time and again at preliminary feedback or peer review stage which now feels unnnecessary and uses up precious reviewer good will.

Using 3.3.0.17

2

Hi @gail,

I don’t have a specific suggestion for your particular use case, no do I necessarily condone the use of AI for this type of pre-screening, but one piece of advice, whatever route you opt to take, would be to be upfront and transparent with authors that their work will be submitted and pre-screened by an LLM. Also, depending on what LLM/AI you end up using, be cognizant of where the data that gets processed by that AI/LLM is processed. This can have implications in terms of author intellectual property and privacy concerns. I haven’t spotted anything that relates specifically to this use case, but COPE has some great resources for journal editors considering AI use (and grappling with generative AI content within submissions):

https://publicationethics.org/guidance/cope-position/authorship-and-ai-tools

-Roger
PKP Team

4

Thanks, @gail for sharing these thoughts, and the value of AI reviews as a prescreening strategy has come up in another discussion I’ve been part of. So without speaking for PKP in any way on the use of AI in scholarly publishing (which is subject to ongoing discussions within PKP), I’d be interested in learning more about how you might consider running a pilot with this idea (which is to say, without formally building into the workflow at this point) with the goal of (a) sparing reviewers of having to review work that is not ready, as you note; (b) assisting editors in making “desk reject” decisions; and (c) aiding authors in preparing their work for peer review. That said, @rcgillis makes good points on what will need to be communicated to authors and other considerations around LLM used. As well, I wouldn’t want to diminish the work of those identifying various threats that AI use is posing to scholarly publishing. It is only to see if, with care and consideration, your idea might offer another way in which AI can advance research.

5

Dear @Willinsky
I am not a native English speaker, Please forgive me if I haven’t expressed myself fully.

There are certain behaviors that are considered contrary to scientific publication ethics, as defined by experts in the field of ethics.
Publication Ethics

Much can be said on this subject, but I am sure that the academic community will conduct genuine scientific studies and make important decisions on this new problem. Currently, we are all, like the rest of the world, following developments regarding the “use of artificial intelligence,” the limits of which are not yet fully defined by laws or ethical rules.

Unless severe academic sanctions are imposed for the dishonest use of artificial intelligence, there is a risk that, just like with the spam email problem, a significant portion of scientific articles submitted to journals will very quickly become fabricated articles written by AI.

Some lazy academics are now having AI prepare fabricated articles that they previously prepared themselves and submitting them to academic scientific journals. As the editor of a Forensic Medicine Journal, I myself have identified one of these articles, although not entirely fabricated, because it contained “fictitious references.” Yes, artificial intelligence can even generate fabricated references. We are currently experiencing a period where editors have to be more careful than ever.

As a former academic who taught “scientific publication ethics” to medical interns for years, I argue that a new concept, "unfair use of artificial intelligence" should be added to that.

Besides the technical aspects of the subject, there is also an academic dimension. Unfortunately, I am not aware of whether we have algorithms or software that can definitively identify such (blurred) articles. Perhaps other colleagues can contribute more on this matter.
Best Regards,
Dr Uğur Koçak

6

Thanks for these responses.

@rcgillis Yes, the risk of people doing this check themselves is that their writing is subsumed into a LLM. This is especially the case for people with only free access to LLM or where they don’t tick the privacy options. But even this is naive in terms of how our data is going to be used in the future. That is why I was considering how we could offer a discreet screening tool for those submitting papers specifically to this journal only to offer them feedback on how well they have met the criteria - not for plagiarism screening. We plan to start using IThenticate for plagiarism.

@willinsky yes you get the potential uses for the editorial process. Any technical ideas welcome. It would need to be done without papers being exposed to feed LLMs. Is there a LLM that can be used “off grid” so it is not connected to LLMs? I’m not a programmer.

@drugurkocak The point I am asking about is a technical matter not about the fabrication of material which is indeed deeply concerning and worthy of ongoing discussion.

7

Hi @gail,

Thanks for clarifying (and thanks to @Willinsky and @drugurkocak for raising some great points). I think, conceivably, such screening might be feasible through the use of a plugin, that is connected to an LLM. Such a plugin would have to be developed and maintained (https://docs.pkp.sfu.ca/dev/plugin-guide/en/) It does remind me of the iThenticate process - but they have their own infrastructure, so this would require hosting something of its own, and there would likely be hosting costs involved . Essentially, you’d need to design a plugin that took in the relevant info, collected author consent, and then submitted it to the LLM (which would be hosted elsewhere). For example, a self-hosted VPS
https://hostsailor.com/blog/self-hosted-llm-vps (not an endorsement of this - it just provides an explanation)

All of this is conceptual and theoretical, plugin and other developers might also wish to comment on how this could be realistically be achieved.

-Roger
PKP Team

8

This is a rather controversial area, but also a growing need in the publishing community. There are some issues involving privacy, since traditionally a submission to a journal was a private communication between editors and authors, and some regard sending the paper to a third party like an LLM service as a violation of this trust. In fact, in the evolution of peer review, authors were initially horrified when editors sent articles to external peer reviewers rather than making the decision themselves. As peer review evolved, it became expected that a third party would see it, but there has always been a risk that the disclosure would result in giving the reviewer inside information to further their work. Some universities have license to use an AI agent under the promise that they won’t train on the inputs, but it’s also the case that university users may be ignorant on the terms of their usage.

Each journal will have their own standards for what they want to screen for, so it’s difficult to imagine a single service that would serve all needs. Among other things, editors might want to check for

  • is it within the scope of the journal?
  • is there an appearance of novelty, or does it repeat something that is already well known?
  • is it written well enough to send to reviewers?
  • does it have sufficient references to ground it in the existing literature?
  • do the authors meet the requirements? (e.g, no group authors or no LLM as an author, or are they real people?)

It’s notable that arxiv has recently instituted some penalties for people who submit articles “If a submission contains incontrovertible evidence that the authors did not check the results of LLM generation”. Examples they give include fabricated bibliographic references or metacomments that were excreted by an LLM, such as “Here is a summary for you”. While it’s easy to imagine software that checks for such basic things, it should almost certainly be used only as a tool to assist the editor in making a decision to desk reject. Our preprint server is now seeing desk rejects on about 50% of submissions, and it has become a burden for the editors so we’re revising our procedures. I assume that others are also experiencing strain on the peer review process.

I am developing something that will perform some basic checks on author reputation, bibliographic references, and checks for obvious falsehoods or plagiarism. It will also institute a moderation process, where authors have to have another author sponsor them to submit their first article. Unfortunately mine will not be written as a plugin to OJS because we use something else. There are some commercial services that one might imagine being used as the basis for a plugin, but there would almost certainly be costs associated with it.

My biggest fear is - what happens if an editor is receiving thousands of papers a day that are generated by AI agents? I recently created a joke website for april fools in which I generated over a thousand bogus articles using a private LLM. I used a machine that is 8 years old, and it was able to generate a new article every 90 seconds. The articles I generated were easily identified as bogus since I did it as a joke, but it’s easy to imagine creating completely plausible articles at mass scale. Given that some authors are already exhibiting evidence of willful fraud, we should be prepared for the fact that some may try to mount such an attack - perhaps to retaliate against a peer review system that they felt treated them unfairly. As the AI agents get become more capable, this potential problem may only get worse. Even if nobody mounts an attack of this type, the tools available to authors makes it easy for them to generate plausible articles at a much higher rate than in the past. It presents a serious threat to the peer review process that we have grown used to in the last 60 years.