New review round while previous review round has not been finished yet

Hey,
something that confused the journal editors we tested OJS with (and ourselves) is why there is a link to create a new review round even though the previous review round has not been finished yet. Is there a setting to prevent that link from showing up (or being active) until the author has submitted a revision of the initial document? And could we then lock round 1 entirely after round 2 has been created?

1 Like

Hi,

this would be of interest to me too. Also, can one delete an ‘empty’ review round? Sometimes, our editors accidentally create multiple review rounds while they really only need one. The other ‘empty’ ones are distracting and it would be nice to get rid of them. With ‘empty’ I mean, that no reviewers have been added to that round.

Hi all,

Thinking about approaches for improving this: it often happens when we impose a hard limit on workflow handling that editors find themselves surprised when they’re prevented from doing something they want. (Most famously, OCS’s hard deadline for preventing new submissions turned out to be too strict in practice, even though in principle a hard deadline sounds reasonable.) So we try to strike a balance that promotes good workflow practice but perhaps stops short of blocking certain kinds of actions.

I’d be hesitant to block reviewers entirely from completing outstanding reviews if they’re already in progress. In part, this is because reviewers have few reasons to finish reviews if they hit roadblocks – they’re generally doing a favour for the journal – and their lives shouldn’t be made harder if they’re going to the trouble of finishing a review.

Perhaps we could prevent editors from starting a new round until the old round has had some kind of “resolution”. For example, the editor might have to explicitly cancel unfinished/unaccepted reviews before starting a new round. In that case, the reviewers would all be notified that the review is no longer needed, rather than trying to come in and perform a review to discover that it’s not possible.

I’m wondering about the risks, though – if a reviewer completes a review on a closed review round, is this bad for the journal?

I do think we should permit the deletion of empty review rounds, so long as they don’t contain any review assignments.

Regards,
Alec Smecher
Public Knowledge Project Team

2 Likes

Hey @asmecher,
I understand your concern. And there may be workflows where it may make sense to have several simultaneous review rounds. This is not the case in the journals we tested with so the editors there just got confused by that option being there. For example, they would click on “Send to Review”, then see the review screen and because there is a link labeled “New Review Round” they would think they would have to initiate the first review round, so they would click that as the very first thing. As they then see the second review round appear, they would be confused as to where that was coming from and which of the two review rounds they should use…

In addition to the ability to have multiple simultaneous review rounds being confusing to the editors, I’m not sure how we should deal with it for the editor plugin. What we have so far is that when a new review round is initiated, it takes the latest document and creates a copy of that to be used in the new review round. This works if the review rounds are all in order. But it falls apart if the editor first creates review round two (which copies the latest document from the previous review round at the time of the review round being initiated), then goes back to review round one in which reviewers are nominated to review the draft and a second revised version is submitted by the author and only then the previously created second review round is used. Here the editor is confused again: Why does the second review round start with the document from before the first review round and not the revised document created by the author at the end of review round one?

What exactly is the purpose of the multiple simultaneous review rounds? Are some journals actually requiring that, or is more that one will let the editor create a new review round in case the reviewers did not respond?

If this covers all your use cases, that would sound good to me and should also create less confusion.

Would this be any different from what happens now? I assume that journals internally ‘close’ review rounds, even if the software doesn’t permit them to. I mean what else can they do if an author has finished the third review round with a third revision, and suddenly he receives a review of the first review round commenting on things that have been removed or changed several round before? Sure, the software currently accepts that outdated review, but will it be acted upon?

Hi @Johannes_Wilm/all,

I know almost nothing about the use cases that our journals use in practice – I’m aware that journals sometimes start a new round by accident, and want to delete it, but can only speculate about how formal the close of the review process is.

I suspect the frequency of publication and mean-time-to-publish will have a major influence: journals that publish frequently will have more pressure to complete the workflow quickly, and closing a review round likely means the next stage has begun and more feedback is going to muddy the waters. Others that publish at a slower pace, or have a buffer of unpublished content ready for scheduling, will probably want to accept review feedback that might fall outside of the review period. (These details will probably also vary by discipline.)

Can you each write a little more about the motivating circumstances? If it’s e.g. just dealing with accidentally created rounds, then I’d rather we tackled that directly.

In my opinion, the best outcome is that using OJS guides editors into a good workflow, but permits recovery from mistakes and outliers. Imposing hard limits on reviewers is especially tricky, as they’re volunteers (and often late).

Regards,
Alec Smecher
Public Knowledge Project Team

Ok, so in our case it is A) to make it clearer to journal editors how to use the ‘create new rounds’ button (you should not click on the ‘create new round’ button when you first enter the review screen, you should not click that button either for each new reviewer, altogether do not click the button ever unless the previous review round is somehow ‘over’) B) to give editors a better intuitive understanding of how the workflow is supposed to work. Right now it does not seem clear that they under normal circumstances should click the “request revisions” button first and the “create new round” button only after such a revision has been submitted.

This could for example be fixed by: A) If there are no reviewers nominated in the current round yet simply disable the ‘create new round’ button entirely. B) If the editor clicks the button before reviews are in present a warning screen: “reviews for the current review round are not yet done. Are you sure you want to go ahead and create a new review round?” and in case it is clicked before the author has resubmitted a warning screen saying: “The author has not yet had the chance to respond to the reviews from the current review round. Are you sue you want to go ahead and create another review round?” That would not directly disable the option of creating review rounds at any given time, yet it would guide editors to follow a useful workflow.

1 Like

We have actually two issues with our journals here.

  1. Accidental new review rounds. I think a warning message like @Johannes_Wilm suggested (“reviews for the current review round are not yet done. Are you sure you want to go ahead and create a new review round?”) would be sufficient for us. This would leave the option open to have two simultanous review rounds, although I can not image a situation where one would need that.

  2. Close a review round although not all reviews have arrived yet. We found that it is not possible to cancel a reviewer once they have agreed to do the review. We have a regular review deadline of 5 weeks. After that we keep reminding and reminding but sometimes the reviewer does just not respond although they agreed at the beginning or they decline because something else came up in the meantime. In OJS2 we unassigned them and sent a polite email to explain the situation. In OJS3.1.0.1, the unassign button disappears once the reviewer agreed to review and there is no way to remove them from the submission. We need that unassign button (in combination with a prepared email) to prevent causing unnecessary work for reviewers when the submission has moved on to the next stage already. So far, we communicated this via email to the reviewer but the editor easily loses oversight who was cancelled and who is active when all reviewers are still listed and the submission status still shows ‘review overdue’.

Hi all,

Thanks, those are both helpful. How does this sound?

  • Add a warning message when the editor tries to create a new round, if reviews in the current round are not all “complete”.
  • Add the ability for editors to cancel reviews that have been accepted, but not yet completed.
  • Add the ability for editors to delete review rounds, if they don’t yet include any review assignments.
  • Otherwise, continue to permit review activity on previous rounds.

Regards,
Alec Smecher
Public Knowledge Project Team

3 Likes

These steps sound fine. How about additionally:

  • Not allowing new rounds if there are no review assignments at all in the current round? Or would there be a reason for why one would request a new review round even though the current round is still empty? The reason this would be important for us is that the editors we tested with were confused by the availability of the button and clicked the “new round” button before doing any assignments.

  • Warn if a new review round is requested even though no revised version has been submitted by the author? Or maybe I’m confused: if there is no revised version of the submission document – why would one assign reviewers in a new review round and not the current one?

  • Don’t permit assigning new reviewers on old review rounds, or why would this be allowed? (This should not stop reviewers who already have been assigned to old review rounds to continue their activities)

Instead of not permitting things, there could alternatively also be warning messages in all those situations, just in case it makes sense to do these things in certain edge case situations.

2 Likes

Hi @asmecher,

sounds good to me. This would solve most of the problems we currently have with the review management.

Hi @asmecher and @Johannes_Wilm ,

one more scenario I have come across recently:

A reviewer declined the first request. Two months later, the editor is still in search of more reviewers for the same submission and asked the reviewer again (via personal contact). The reviewer agreed but the editor is now unable to add the reviewer again to the first round. He has to create a second round to add them although the first round is not finished. It would be nice if one could re-assign a reviewer who initially declined. Otherwise this would be a valid scenario where one would really need two active review rounds in parallel. If two rounds are active with one reviewer each, the main submission list shows only one reviewer assigned, though. Maybe this is only the count from the latest round?

1 Like

Great proposal, it would solve our problems mainly, and with the proposal of Johannes_Wilm, entirely. :smiley:

Thanks!
Bogi

Hi all,

I’ve filed this for attention at Improve "New Review Round" behavior · Issue #3585 · pkp/pkp-lib · GitHub.

Regards,
Alec Smecher
Public Knowledge Project Team

Just asking if this was updated in the latest release, or is it planned for sometime soon?

@haydenyoung

Hi @NadineWubbeling,

Not yet – the best way to stay current is to watch the github issue linked above for changes.

Regards,
Alec Smecher
Public Knowledge Project Team

I agree with your proposal of additions. With these changes put in place, the review work flow will definitely improve!

Hi,

We are using OJS 3.1.2-1 at University of Bordeaux for several journals. Our main journal encounter a similar problem. Is it planned for a next release ?

Thanks in advance four your answer.
Best regards.
Helene