OJS 3.4.0.1 - PKP\jobs\doi\DepositSubmission has been attempted too many times or run too long

Software Support
1

Hello everyone,

OJS: 3.4.0.1
PHP: 8.1

Today, I noticed a following error in error log, need your guide to sovle this issue…

[27-Jul-2023 13:16:06 UTC] Illuminate\Queue\MaxAttemptsExceededException: PKP\jobs\doi\DepositSubmission has been attempted too many times or run too long. The job may have previously timed out. in /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/Worker.php:746
Stack trace:
#0 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(505): Illuminate\Queue\Worker->maxAttemptsExceededException()
#1 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(415): Illuminate\Queue\Worker->markJobAsFailedIfAlreadyExceedsMaxAttempts()
#2 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(375): Illuminate\Queue\Worker->process()
#3 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(326): Illuminate\Queue\Worker->runJob()
#4 /home/seisense/journal.seisense.com/lib/pkp/classes/core/PKPQueueProvider.php(104): Illuminate\Queue\Worker->runNextJob()
#5 /home/seisense/journal.seisense.com/lib/pkp/classes/queue/JobRunner.php(220): PKP\core\PKPQueueProvider->runJobInQueue()
#6 /home/seisense/journal.seisense.com/lib/pkp/classes/task/ProcessQueueJobs.php(68): PKP\queue\JobRunner->processJobs()
#7 /home/seisense/journal.seisense.com/lib/pkp/classes/scheduledTask/ScheduledTask.php(165): PKP\task\ProcessQueueJobs->executeActions()
#8 /home/seisense/journal.seisense.com/plugins/generic/acron/AcronPlugin.php(271): PKP\scheduledTask\ScheduledTask->execute()
#9 /home/seisense/journal.seisense.com/plugins/generic/acron/AcronPlugin.php(186): APP\plugins\generic\acron\AcronPlugin->_shutdownFunction()
#10 [internal function]: APP\plugins\generic\acron\AcronPlugin->APP\plugins\generic\acron{closure}()
#11 {main}
[27-Jul-2023 13:16:06 UTC] #0 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(505): Illuminate\Queue\Worker->maxAttemptsExceededException()
#1 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(415): Illuminate\Queue\Worker->markJobAsFailedIfAlreadyExceedsMaxAttempts()
#2 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(375): Illuminate\Queue\Worker->process()
#3 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(326): Illuminate\Queue\Worker->runJob()
#4 /home/seisense/journal.seisense.com/lib/pkp/classes/core/PKPQueueProvider.php(104): Illuminate\Queue\Worker->runNextJob()
#5 /home/seisense/journal.seisense.com/lib/pkp/classes/queue/JobRunner.php(220): PKP\core\PKPQueueProvider->runJobInQueue()
#6 /home/seisense/journal.seisense.com/lib/pkp/classes/task/ProcessQueueJobs.php(68): PKP\queue\JobRunner->processJobs()
#7 /home/seisense/journal.seisense.com/lib/pkp/classes/scheduledTask/ScheduledTask.php(165): PKP\task\ProcessQueueJobs->executeActions()
#8 /home/seisense/journal.seisense.com/plugins/generic/acron/AcronPlugin.php(271): PKP\scheduledTask\ScheduledTask->execute()
#9 /home/seisense/journal.seisense.com/plugins/generic/acron/AcronPlugin.php(186): APP\plugins\generic\acron\AcronPlugin->_shutdownFunction()
#10 [internal function]: APP\plugins\generic\acron\AcronPlugin->APP\plugins\generic\acron{closure}()
#11 {main}
[27-Jul-2023 13:16:17 UTC] #0 /home/seisense/journal.seisense.com/lib/pkp/jobs/doi/DepositSubmission.php(63): APP\plugins\generic\crossref\CrossrefPlugin->depositSubmissions()
#1 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): PKP\jobs\doi\DepositSubmission->handle()
#2 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\Container\BoundMethod::Illuminate\Container{closure}()
#3 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\Container\Util::unwrapIfClosure()
#4 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\Container\BoundMethod::callBoundMethod()
#5 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Container/Container.php(661): Illuminate\Container\BoundMethod::call()
#6 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(128): Illuminate\Container\Container->call()
#7 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(141): Illuminate\Bus\Dispatcher->Illuminate\Bus{closure}()
#8 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(116): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}()
#9 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(132): Illuminate\Pipeline\Pipeline->then()
#10 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(124): Illuminate\Bus\Dispatcher->dispatchNow()
#11 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(141): Illuminate\Queue\CallQueuedHandler->Illuminate\Queue{closure}()
#12 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(116): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}()
#13 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(126): Illuminate\Pipeline\Pipeline->then()
#14 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(70): Illuminate\Queue\CallQueuedHandler->dispatchThroughMiddleware()
#15 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(98): Illuminate\Queue\CallQueuedHandler->call()
#16 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(425): Illuminate\Queue\Jobs\Job->fire()
#17 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(375): Illuminate\Queue\Worker->process()
#18 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(326): Illuminate\Queue\Worker->runJob()
#19 /home/seisense/journal.seisense.com/lib/pkp/classes/core/PKPQueueProvider.php(104): Illuminate\Queue\Worker->runNextJob()
#20 /home/seisense/journal.seisense.com/lib/pkp/classes/queue/JobRunner.php(220): PKP\core\PKPQueueProvider->runJobInQueue()
#21 /home/seisense/journal.seisense.com/lib/pkp/classes/task/ProcessQueueJobs.php(68): PKP\queue\JobRunner->processJobs()
#22 /home/seisense/journal.seisense.com/lib/pkp/classes/scheduledTask/ScheduledTask.php(165): PKP\task\ProcessQueueJobs->executeActions()
#23 /home/seisense/journal.seisense.com/plugins/generic/acron/AcronPlugin.php(271): PKP\scheduledTask\ScheduledTask->execute()
#24 /home/seisense/journal.seisense.com/plugins/generic/acron/AcronPlugin.php(186): APP\plugins\generic\acron\AcronPlugin->_shutdownFunction()
#25 [internal function]: APP\plugins\generic\acron\AcronPlugin->APP\plugins\generic\acron{closure}()
#26 {main}
[27-Jul-2023 13:16:18 UTC] #0 /home/seisense/journal.seisense.com/lib/pkp/jobs/doi/DepositSubmission.php(63): APP\plugins\generic\crossref\CrossrefPlugin->depositSubmissions()
#1 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): PKP\jobs\doi\DepositSubmission->handle()
#2 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\Container\BoundMethod::Illuminate\Container{closure}()
#3 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\Container\Util::unwrapIfClosure()
#4 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\Container\BoundMethod::callBoundMethod()
#5 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Container/Container.php(661): Illuminate\Container\BoundMethod::call()
#6 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(128): Illuminate\Container\Container->call()
#7 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(141): Illuminate\Bus\Dispatcher->Illuminate\Bus{closure}()
#8 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(116): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}()
#9 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(132): Illuminate\Pipeline\Pipeline->then()
#10 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(124): Illuminate\Bus\Dispatcher->dispatchNow()
#11 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(141): Illuminate\Queue\CallQueuedHandler->Illuminate\Queue{closure}()
#12 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(116): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}()
#13 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(126): Illuminate\Pipeline\Pipeline->then()
#14 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(70): Illuminate\Queue\CallQueuedHandler->dispatchThroughMiddleware()
#15 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(98): Illuminate\Queue\CallQueuedHandler->call()
#16 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(425): Illuminate\Queue\Jobs\Job->fire()
#17 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(375): Illuminate\Queue\Worker->process()
#18 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(326): Illuminate\Queue\Worker->runJob()
#19 /home/seisense/journal.seisense.com/lib/pkp/classes/core/PKPQueueProvider.php(104): Illuminate\Queue\Worker->runNextJob()
#20 /home/seisense/journal.seisense.com/lib/pkp/classes/queue/JobRunner.php(220): PKP\core\PKPQueueProvider->runJobInQueue()
#21 /home/seisense/journal.seisense.com/lib/pkp/classes/task/ProcessQueueJobs.php(68): PKP\queue\JobRunner->processJobs()
#22 /home/seisense/journal.seisense.com/lib/pkp/classes/scheduledTask/ScheduledTask.php(165): PKP\task\ProcessQueueJobs->executeActions()
#23 /home/seisense/journal.seisense.com/plugins/generic/acron/AcronPlugin.php(271): PKP\scheduledTask\ScheduledTask->execute()
#24 /home/seisense/journal.seisense.com/plugins/generic/acron/AcronPlugin.php(186): APP\plugins\generic\acron\AcronPlugin->_shutdownFunction()
#25 [internal function]: APP\plugins\generic\acron\AcronPlugin->APP\plugins\generic\acron{closure}()
#26 {main}
[27-Jul-2023 13:16:18 UTC] #0 /home/seisense/journal.seisense.com/lib/pkp/jobs/doi/DepositSubmission.php(63): APP\plugins\generic\crossref\CrossrefPlugin->depositSubmissions()
#1 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): PKP\jobs\doi\DepositSubmission->handle()
#2 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\Container\BoundMethod::Illuminate\Container{closure}()
#3 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\Container\Util::unwrapIfClosure()
#4 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\Container\BoundMethod::callBoundMethod()
#5 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Container/Container.php(661): Illuminate\Container\BoundMethod::call()
#6 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(128): Illuminate\Container\Container->call()
#7 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(141): Illuminate\Bus\Dispatcher->Illuminate\Bus{closure}()
#8 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(116): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}()
#9 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(132): Illuminate\Pipeline\Pipeline->then()
#10 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(124): Illuminate\Bus\Dispatcher->dispatchNow()
#11 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(141): Illuminate\Queue\CallQueuedHandler->Illuminate\Queue{closure}()
#12 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(116): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}()
#13 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(126): Illuminate\Pipeline\Pipeline->then()
#14 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(70): Illuminate\Queue\CallQueuedHandler->dispatchThroughMiddleware()
#15 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(98): Illuminate\Queue\CallQueuedHandler->call()
#16 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(425): Illuminate\Queue\Jobs\Job->fire()
#17 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(375): Illuminate\Queue\Worker->process()
#18 /home/seisense/journal.seisense.com/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(326): Illuminate\Queue\Worker->runJob()
#19 /home/seisense/journal.seisense.com/lib/pkp/classes/core/PKPQueueProvider.php(104): Illuminate\Queue\Worker->runNextJob()
#20 /home/seisense/journal.seisense.com/lib/pkp/classes/queue/JobRunner.php(220): PKP\core\PKPQueueProvider->runJobInQueue()
#21 /home/seisense/journal.seisense.com/lib/pkp/classes/task/ProcessQueueJobs.php(68): PKP\queue\JobRunner->processJobs()
#22 /home/seisense/journal.seisense.com/lib/pkp/classes/scheduledTask/ScheduledTask.php(165): PKP\task\ProcessQueueJobs->executeActions()
#23 /home/seisense/journal.seisense.com/plugins/generic/acron/AcronPlugin.php(271): PKP\scheduledTask\ScheduledTask->execute()
#24 /home/seisense/journal.seisense.com/plugins/generic/acron/AcronPlugin.php(186): APP\plugins\generic\acron\AcronPlugin->_shutdownFunction()
#25 [internal function]: APP\plugins\generic\acron\AcronPlugin->APP\plugins\generic\acron{closure}()
#26 {main}

Thanks
seisense

2

I am witnessing a plenty of failed job instances (see screenshot) which seems are relevent to this error.

image
image2001×1540 250 KB

3

Hi everyone,
I have a total 23 articles from the first two issues of my journal where DOI is assigned from Zenodo. I believe Acron job is trying to deposit these submission to crossref and thats where the job is failed. So, I can safely ignore and delete from the job queue.

Thanks
seisense

4

HI @seisense , can you please provide some details how the jobs are getting executed in your system ? Are you using the JobRunner with the Acron or Cron or Worker Thread ? if worker thread, then what is the configuration of Supervisor ?

This is basically happening when job failed and then picked up too soon or immediately again to get executed again. Now when the condition such as

JOB_MAX_TRIES * JOB_BACKOFF_TIME_IN_SECONDS >= JOB_RETRY_UNTIL_IN_SECONDS

satisfy, it will cause such issue . to overcome such section, it’s possible to use the worker thread to set such condition as

php lib/pkp/tools/jobs.php work --tries=3 --backoff=1

As per our doc, we strongly suggest to use worker thread to handle jobs for production env . However I will create an issue to set few default values in the base job class .

1 Like
5

Hi @touhidur , we are using OJS built in Acron and havn’t setup any other job runner.
Thanks
seisense

6

@seisense In that case I can only think of the situation where the job runner picked up failed job immediately and this happened continuously which cause that error . we will add few default config value to BaseJob as a safeguard and it will be included in next 3.4.0-2 release . However I strongly suggest you to set up the worker thread with supervisor to handle the jobs rather than relying on JobRunner . Take a look here at https://docs.pkp.sfu.ca/admin-guide/en/deploy-jobs .

Regards
PKP Team

1 Like