[OJS 3.0.2] CrossRef XML Export Plugin: Status not refreshed properly

ojs3

#11

@bozana, thanks for answer!

{“status”:“ok”,“message-type”:“deposit-list”,“message-version”:“1.0.0”,“message”:{“items-per-page”:20,“query”:{“start-index”:0,“search-terms”:null},“total-results”:1,“items”:[{“handoff”:{“status”:“completed”,“delay-millis”:2718.2818284590453,“try-count”:1,“timestamp”:1542705173372},“dois”:[“10.32420/1996.1”,“10.32420/1996.1.15”,“10.32420/1996.1”,“10.32420/1996.1.14”,“10.32420/1996.1”,“10.32420/1996.1.13”,“10.32420/1996.1”,“10.32420/1996.1.12”,“10.32420/1996.1”,“10.32420/1996.1.11”,“10.32420/1996.1”,“10.32420/1996.1.10”,“10.32420/1996.1”,“10.32420/1996.1.8”],“parent”:null,“matched-citation-count”:0,“filename”:null,“submitted-at”:“Tue Nov 20 04:12:53 EST 2018”,“status”:“submitted”,“length”:19558,“content-type”:“application/vnd.crossref.deposit+xml”,“pingback-url”:null,“citation-count”:0,“test”:false,“owner”:“uarr”,“batch-id”:“fe660ba3-9df1-4956-a968-2d401661b40e”}]}}

thats what i receive


#12

Hi @redukr

I think this is again at the Crossref site – the status for that submission seems not to be updated at the Crossref site/deposit API yet – s. the part ...“submitted-at”:“Tue Nov 20 04:12:53 EST 2018”,“status”:“submitted”... i.e. the status seems to be “submitted” here.
One another thing that I am wondering about: it seems like you have submitted it today, Nov 20th ? Then maybe to wait a little bit and check later or latest tomorrow…
If it is on the Crossref site, maybe you could contact the Crossref support and explain that you have submitted it and the DOIs seem to be active, but that this deposit API does not update the status…

Best,
Bozana


#13

Hi @bozana I receive answer from crossref, but i amnot sure that i understand what i should wrote to them:

"Hello,
Many thanks for your email.
The DOI is showing as being registered with us as you can see from the metadata here: http://doi.crossref.org/search/doi?pid=support@crossref.org&format=unixsd&doi=10.32420%2F1996.1.12
Can I ask you to elaborate further on your query and I will try and help you with this.
Best regards,
Paul Davis
Product Support Specialist
@CrossrefSupport
"


#14

Hi @redukr

Hmmm… Did you write them that the status of the last deposit is still “submitted” when using the Crossref deposit API i.e. https://api.crossref.org/deposits?filter=doi:10.32420/1996.1.12 ? – Because the support person is pointing to another API call i.e. not mentioning the deposit API at all…

Best,
Bozana


#15

@bozana i have ticked opened for 6 days, at crossref site. no answer…


#16

@bozana i receive answer from crossref. as i understand, that’s problem at crossref side:

"I will pass the query regarding that API call onto our developers and see what comes back.
I will keep you posted on any progress from this and once again apologies for the delay. "


#17

Hi @redukr,

Yes, lets see what they say i.e. if they can do something to actualize the statuses via that deposit API…

Best,
Bozana


#18

We have the same problem with thousands of DOIs. Any update on this?

It is also probably making our scheduled DOI task very slow. The task is trying to register / set a new status for all those thousands of articles that have the “submitted” status and does this every day.


#19

Also, what is the correct crossref::status for an active DOI?

Seems that “active” is just a label, so is it “registered” or “found”?


#20

Ok so @bozana if I am understanding the Crossref export code correctly, this request here https://github.com/pkp/ojs/blob/master/plugins/importexport/crossref/CrossrefInfoSender.inc.php#L63 should only return articles/DOIs that have crossref::status set to EXPORT_STATUS_NOT_DEPOSITED which is basically notDeposited, right?

But the thing is that the filter is not working. It seems the request is returning all articles that have a DOI regardless of the status they have. This means that every day all of the DOIs in the system are registered again.

edit:
the EXPORT_STATUS_NOT_DEPOSITED is not visible in the params here: https://github.com/pkp/ojs/blob/master/classes/article/PublishedArticleDAO.inc.php#L782

edit2:
seems that $pubIdSettingValue does not have a value at all here: https://github.com/pkp/ojs/blob/master/classes/article/PublishedArticleDAO.inc.php#L754

edit3: and that is because this setting here https://github.com/pkp/ojs/blob/master/classes/plugins/DOIPubIdExportPlugin.inc.php#L138 overrides the setting here https://github.com/pkp/ojs/blob/master/classes/plugins/PubObjectsExportPlugin.inc.php#L425
But is that intentional?

edit4: also even if the EXPORT_STATUS_NOT_DEPOSITED param is passed to getExportable from the crossref plugin, the sql that is created does not include it. The logic that creates the sql leaves that out. This would work:

		import('classes.plugins.PubObjectsExportPlugin');
		if ($pubIdSettingName && $pubIdSettingValue && $pubIdSettingValue == EXPORT_STATUS_NOT_DEPOSITED) {
			$params[] = $pubIdSettingValue;
		}

		$result = $this->retrieveRange(
			'SELECT	s.*, ps.*,
				' . $this->getFetchColumns() . '
			FROM	published_submissions ps
				JOIN issues i ON (ps.issue_id = i.issue_id)
				LEFT JOIN submissions s ON (s.submission_id = ps.submission_id)
				' . ($pubIdType != null?' LEFT JOIN submission_settings ss ON (s.submission_id = ss.submission_id)':'')
				. ($title != null?' LEFT JOIN submission_settings sst ON (s.submission_id = sst.submission_id)':'')
				. ($author != null?' LEFT JOIN authors au ON (s.submission_id = au.submission_id)':'')
				. ($pubIdSettingName != null?' LEFT JOIN submission_settings sss ON (s.submission_id = sss.submission_id AND sss.setting_name = ?)':'')
				. ' ' . $this->getFetchJoins() .'
			WHERE
				i.published = 1 AND s.context_id = ? AND s.status <> ' . STATUS_DECLINED
				. ($pubIdType != null?' AND ss.setting_name = ? AND ss.setting_value IS NOT NULL':'')
				. ($title != null?' AND (sst.setting_name = ? AND sst.locale = ? AND sst.setting_value LIKE ?)':'')
				. ($author != null?' AND (au.first_name LIKE ? OR au.middle_name LIKE ? OR au.last_name LIKE ?)':'')
				. ($issueId != null?' AND ps.issue_id = ?':'')
				. (($pubIdSettingName != null && $pubIdSettingValue != null && $pubIdSettingValue != EXPORT_STATUS_NOT_DEPOSITED)?' AND sss.setting_value IS NULL':'')
				. (($pubIdSettingName != null && $pubIdSettingValue != null && $pubIdSettingValue == EXPORT_STATUS_NOT_DEPOSITED)?' AND sss.setting_value = ?':'')
				. (($pubIdSettingName != null && is_null($pubIdSettingValue))?' AND (sss.setting_value IS NULL OR sss.setting_value = \'\')':'')
			. ' ORDER BY ps.date_published DESC, s.submission_id DESC',
			$params,
			$rangeInfo
		);

But I am not sure whether I am just misunderstandin the whole logic here.

In any case, the current code tries to register all created DOIs in the system every day, or at least is trying to check the status of all DOIs. With larger installations this means cron processes that can run for several hours, probably even days.

I understand that there could be a need to register articles even when they are already registered once. I mean you need to update the status and the changes in the metadata. But I am not sure if the current situation where all DOIs are reregistereted all the time is intended to do this?


#21

Hi @ajnyga

The function getUnregisteredArticles will look after all articles that does not have a setting ‘crossref::registeredDoi’ i.e. all not successfully deposited and active article DOIs. That setting will only be set/inserted into the DB when the DOI is active i.e. when the last registration/deposit was successfully completed and the DOI metadata can be found in the crossref metadata.

EDIT: the function _getObjectsToBeDeposited (https://github.com/pkp/ojs/blob/master/plugins/importexport/crossref/CrossrefInfoSender.inc.php#L125) then filters those unregistered articles (returned from getUnregisteredArticles): it looks after the new status i.e. updates the status and returns only those that are not deposited (s. https://github.com/pkp/ojs/blob/master/plugins/importexport/crossref/CrossrefInfoSender.inc.php#L129-L135) i.e. that have no ‘crossref::status’ setting name.

The setting value for the setting name ‘crossref::status’ in the DB will be ‘found’ for successfully deposited (last deposit attempt) and active article DOIs.

Does this help?
Best,
Bozana


#22

It is also probably making our scheduled DOI task very slow. The task is trying to register / set a new status for all those thousands of articles that have the “submitted” status and does this every day.

Yes, that could be – this is probably the updateDepositStatus here https://github.com/pkp/ojs/blob/master/plugins/importexport/crossref/CrossrefInfoSender.inc.php#L129. This code line is needed in order to see if the status has been changed to ‘failed’ and to notify the user in that case.

And now I see the wrong logic there, the line https://github.com/pkp/ojs/blob/master/plugins/importexport/crossref/CrossrefInfoSender.inc.php#L131 should go before calling updateDepositStatus and https://github.com/pkp/ojs/blob/master/plugins/importexport/crossref/CrossrefInfoSender.inc.php#L132-L135 should go after https://github.com/pkp/ojs/blob/master/plugins/importexport/crossref/CrossrefInfoSender.inc.php#L137, and there the $newStatus instead of $currentStatus should be checked.
But, I think this would not change how often the updateDepositStatus has been called.


#23

Hi @bozana

Thanks, I figured over the weekend that it is not just the DOIs with submitted status that are being reprocessed but basically all DOIs with any status.

I am just wondering whether there is a need to keep updating the deposit status of DOIs that have “found” status and are attached to articles that have not been edited lately.

The fact that API is now not returning the correct status is of course a separate issue here and will be hopefully fixed. But when it is fixed it would be maybe good both for us and the Crossref API that the updateDepositStatus is called when it is actually needed.

Or is the logic here that OJS needs to know if a DOI that used to work is changed to failed status? I would thinkt that in those cases Crossref will contact the journal in any case? At least I get a lot of conflict reports etc. straight from Crossref.


#24

Hi @ajnyga

The DOIs with status ‘found’ should not be considered in the scheduled task – they should have the setting ‘crossref::registeredDoi’ so the function getUnregisteredArticles would not return them. Thus I am wondering why is this happening for you :open_mouth: :thinking: Do those articles have that setting_name? Or is there maybe a bug in the system? :thinking:
Else, all other DOIs need an automatic status update.
(If an active DOI is manually (this does not happen automatically) re-deposited, e.g. because some metadata have been changed, the old settings will be removed and the status of that last deposit will be considered).

Thanks!
Bozana


#25

Ok, do not waste time on this now! Could very well be that the one’s with found are not among the ones being registestered (looking at the way I was debugging this). Only very few of our 4000 DOIs have that status. About 98% have status submitted. So the original issue here is clearly the important one.

But, would be nice to have OJS reregister DOIs for articles that have been modified. Not talking only about revisions when they come, but maybe just update the metadata whenever the submissions timestamp has been touched or the metadataform saved? Most articles are not ever touched afterwards unless there is a real reason.


#26

^ following this as well.

Our use case is that we moved our registration provider from EZID back to Crossref, and based on enhancements in the plugin (e.g. adding datamining URLs, correcting pagination metadata, etc.) we had interest in resupplying metadata at-large.

Now that I’m looking at this in more detail, it appears that the API endpoint we are using (https://api.crossref.org/deposits) is not documented here:


and is not mentioned here:

Have we checked with Crossref regarding the status (perhaps legacy or EOL?) of the endpoint we’re currently using? If not, I can do so.

Edit: found the deprecated documentation:

and inquired about support plans.


#27

I think Bozana is working with the new API already? Not sure about the schedule though.


#28

Crossref confirmed that the “deprecated” API is currently supported with no current plans toward end-of-life.

I can confirm that this bug in the Crossref API also affects OJS 2.4.8, preventing the submissions from ever progressing to “complete” and thus preventing the insertion of the “crossref::registeredDoi” setting.

I’m going to raise this with Crossref support as well.


#29

Hi @ctgraham

Thanks a lot for the information!
I think the integration of the new Crossref deposit API will come with the next 3.1.2 release. I will double check with Crossref that this is OK next week.

Best,
Bozana


#30

@bozana i receive answer:

Paul Davis (Crossref)

Jan 16, 12:55 EST

Hello Georgii,

Apologies for the delay I have been working through various channels to obtain some information for you. Our director of operations has since emailed Bozana as she will be taking the lead on this.

I believe the update with the OJS plugin 3.0.2 obtains the status differently than the older versions of the plugin. Status should now be instant.

If you have any other questions regarding this then please do let me know and once again apologies for the delay.

Best regards,