OJS Support for group-based review process

Describe the problem you would like to solve

At the ASRHE journal we have been operating a group-based review approach for the last four years (see information under ‘Peer Review Process’. We have handled the group-based review processed by defining new roles in OJS and using external tools (email, Doodle) to complement OJS functionality. This ‘manual’ form of organisation is possible but time consuming and not transparent to authors.

Describe the solution you’d like

We would like an additional from of peer review supported in OJS. At the core of the new functionality sits a different perspective of reviewer activity: Instead of working in isolation (as supported by the OJS Reviewer role) the reviewer works in collaboration with the other reviewers and the assigned editor (in our terminology the ‘review group leader’) to conduct the review and collate feedback for the authors.

Who is asking for this feature?

This new feature is required by editors to seamlessly implement group-based peer review. We are confident of the value of the group-based review approach for our authors and for reviewer development (see our open access publication in the International Journal for Academic Development and suggest adding a group-based review option to journals as a way of addressing the shortage of qualified reviewers many journals experience.

Additional information

We have created a document outlining the requirements for supporting the group-based review approach and providing a comparison to the current OJS review approach.

We’d welcome any discussions from implementation or editorial perspectives.

On behalf of the ASRHE Editors, Eva Heinrich

I am a librarian working with a student journal at my institution who is implementing this type of group-based review. I would love to see this type of review supported with reviewers able to connect with one another directly instead of through the editor.

I had a similar need. If you look at the OJS publishing workflow, you can see that the workflow can be thought of as having several phases (submission, reviewing, copy editing, production, and publishing). There is a bit of documentation about how to use OJS for only part of the workflow, but it’s pretty limited.

We (iacr.org) use a separate submission and group review process for two of our journals based on a modification of the HotCRP open source system. HotCRP is also used by ACM, Usenix, and IEEE. It is primarily intended for conferences, but can be used for a journal review system in which papers are submitted periodically by a deadline (e.g., quarterly) and all papers for that deadline are reviewed by a group of reviewers. As a rough measure of the complexity, HotCRP is about 160K lines of PHP, whereas OJS is 1.4M lines of PHP. In reality, HotCRP is quite complicated because it is written in a very terse style of code. We don’t take submissions through OJS - we take them through HotCRP. The output from HotCRP can be thought of as a set of authorizations for authors to upload their post-review versions of accepted papers.

The review process of HotCRP sounds quite a bit different from what you want, but this illustrates the value of making the steps of a publishing workflow into pluggable components. There is quite a bit of experimentation going on in the peer review part of publishing, so it would be nice to be able to extend OJS in this way. There are several other systems that implement a peer review system (e.g., easychair, CMT, openreview.net, etc). openreview.net is used for the NeurIPS conference, which received 12,343 submissions in 2023. :flushed: As far as I know, only HotCRP is open source but you still might be able to use another review system as a black box.

The documentation suggests using the “quick submit” plugin to enter papers into OJS if you are only using it for publishing. We find that approach to be time-consuming since it involves a lot of data entry (it would be completely infeasible for NeurIPS). It seems theoretically possible that you could use the REST API to bypass the review process and create submissions based on the output of HotCRP. I have not used that API and it looks like it would be quite a bit of work. Apparently author agreement to copyright assignment is part of the submission process, but HotCRP does not support that and you would have to build something for that (we built it into our extended version of HotCRP).

I suspect that turning OJS into a more modular system where you can replace the reviewing or copy editing modules would take quite a bit of engineering effort. For our newest journal, we ended up building completely different alternative to OJS. This was motivated in part by our dependence on LaTeX for the copy editing and production, the difficulty of using OJS for only part of the publishing process, and our desire to support richer metadata about publications (e.g, multiple affiliations, more detail than just ROR for affiliations, funding support, unauthenticated ORCID, better support for mathematics in titles and abstracts, import of citations via BibTeX, etc). The publishing practices in computer science are quite a bit different from other disciplines, so I’m not sure how much demand there is for the features we wanted. OJS hits a sweet spot for a lot of academic publishing, but it wasn’t flexible enough for us and didn’t seem to be evolving fast enough. In fairness, they can only address so much with the engineering resources they have and they need to prioritize based on the demand.

Hi @eheinric,

That’s an interesting model; I haven’t encountered it yet – even conventional open reviews are very much based on editor-to-reviewer communication rather than groups.

The closest match to what you’re describing in OJS is the Review Discussion tool, which is available at the bottom of the review workflow page. Make sure that your reviewers are assigned with a review type of Open:
image

…then you’ll be able to create a new discussion including a mix of authors and reviewer as you like:

You can streamline this further by using Settings > Workflow > Emails > Add and edit templates to define prefab email templates that can be selected to kick off the process. (We plan to refine this in the future to allow the templates to be designated for use by certain users, among other things.)

I think the issue you’ll run into using this is that discussions are not considered peer reviews, so some of the tracking and stats of OJS will not include them. The quickest/easiest work around for this would be to have reviewers file a recommendation at the conclusion of the process using the usual peer review tool.

(@kmccurley, OJS is about 300k LOC, not 1.5M – I think you’re counting 3rd party dependencies like Laravel. And about 230k of the 300k is shared by OMP and OPS. So while we do have to contend with complexity, it’s not nearly as bad as that!)

Thanks,
Alec Smecher
Public Knowledge Project Team

Hi all,
Thank you for your interest in this topic (and apologies for my late reply … end of semester marking).

@kshuttle would you like to catch up for more information on how we are handling group-based review for ASRHE in OJS at the moment? (e.heinrich@massey.ac.nz)

@asmecher Using the Review Type Open is an interesting idea I had not considered before. It would be ok for our journal (we share reviewer names with our authors by adding those manually to the feedback at the moment), yet other journals might stay with anonymous reviewers despite using a group-based review approach.
Using ‘Open’ and reviewers assigned via the standard OJS mechanism would allow us to do the same we are currently doing with our review group member/leaders roles based on Section Editor - manually setting up a discussion (carefully avoiding adding the author) for our discussions around compiling the feedback for the authors following our review group meetings.
We still had the issue around not using our reviewers in a way designed for in OJS. We could ask them to submit their feedback as ‘reviews’ (would those be shared with the author under ‘Open’?) to satisfy the system that the reviewers have done their jobs but this would not sit naturally with our cooperative group-based approach.
Also, a journal might want to mix traditional and group-based reviews (independent of using open or anonymous). This would be valuable in the context of reviewer training which is a strength of the group-based approach that allows integrating less experienced reviewers in review discussions. Having different review modes one can both allow on journal level would allow selection on a per article basis (different from the current Review Type settings that allow for only one option).

Kind Regards, Eva