Native XML plugin fails on exporting journal issues

dh_its · June 2, 2020, 7:38am

Hi all, just wondering if you have any ideas how to resolve this issue I have had with the Native XML import/export plugin.

Some details about our installation:

We are currently on OJS 3.1.2.1 (we have not yet upgraded to 3.2, and probably won’t for a while yet due to the complexity of the upgrade process)
This is a multi-journal installation of OJS
The data dates from OJS 2.0 or thereabouts - we have upgraded throughout the years
We are using PHP 7.2.10 (with an upgrade to 7.4 scheduled later this year) with MySQL, running the site in a Docker container on Google Cloud Platform (probably not relevant but it gives you an idea of our setup)
We are currently in the process of migrating one of the journals away from our hosting infrastructure and this is a part of the export process

The issue is that we get a 502 bad gateway error in the GUI when attempting to export a journal’s issue. Running the export in the CLI results in the below:

Validation errors:
Element '{http://pkp.sfu.ca}issues': Missing child element(s). Expected is ( {http://pkp.sfu.ca}issue ).
Invalid XML:
<?xml version="1.0"?>
<issues xmlns="http://pkp.sfu.ca" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://pkp.sfu.ca native.xsd"/>

<h1>Could not convert selected objects.</h1>ojs2: Could not convert selected objects.

There is another problem related to this, but with a journal issue dating from 2014 - the script hangs until it is killed automatically.

PHP Notice:  Only variables should be passed by reference in /opt/app-root/src/plugins/importexport/native/filter/IssueNativeXmlFilter.inc.php on line 218
PHP Notice:  Only variables should be passed by reference in /opt/app-root/src/plugins/importexport/native/filter/IssueNativeXmlFilter.inc.php on line 197
PHP Warning:  Declaration of SubmissionKeywordEntryDAO::getByControlledVocabId($controlledVocabId, $rangeInfo = NULL) should be compatible with ControlledVocabEntryDAO::getByControlledVocabId($controlledVocabId, $rangeInfo = NULL, $filter = NULL) in /opt/app-root/src/lib/pkp/classes/submission/SubmissionKeywordEntryDAO.inc.php on line 20
PHP Warning:  Declaration of SubmissionAgencyEntryDAO::getByControlledVocabId($controlledVocabId, $rangeInfo = NULL) should be compatible with ControlledVocabEntryDAO::getByControlledVocabId($controlledVocabId, $rangeInfo = NULL, $filter = NULL) in /opt/app-root/src/lib/pkp/classes/submission/SubmissionAgencyEntryDAO.inc.php on line 44
PHP Warning:  Declaration of SubmissionDisciplineEntryDAO::getByControlledVocabId($controlledVocabId, $rangeInfo = NULL) should be compatible with ControlledVocabEntryDAO::getByControlledVocabId($controlledVocabId, $rangeInfo = NULL, $filter = NULL) in /opt/app-root/src/lib/pkp/classes/submission/SubmissionDisciplineEntryDAO.inc.php on line 44
PHP Warning:  Declaration of SubmissionSubjectEntryDAO::getByControlledVocabId($controlledVocabId, $rangeInfo = NULL) should be compatible with ControlledVocabEntryDAO::getByControlledVocabId($controlledVocabId, $rangeInfo = NULL, $filter = NULL) in /opt/app-root/src/lib/pkp/classes/submission/SubmissionSubjectEntryDAO.inc.php on line 44
Killed

I can export articles using the same script just fine, so I think there’s a data/config issue but not sure where to look.

Any ideas?

Thanks,
Daniel

asmecher · June 2, 2020, 6:08pm

Hi @dh_its,

The Killed message suggests the script encountered a timeout or memory limit. I’d suggest checking your logs (e.g. PHP-FPM logs, or Apache logs, or similar) to see if those indicate which, and temporarily trying to relax those limits to see if it helps (particularly the memory limit). You might have better luck exporting via the command line as limits are frequently more lenient there.

Regards,
Alec Smecher
Public Knowledge Project Team

dh_its · June 3, 2020, 12:12am

Hi Alec,

I’ve been running the scripts in CLI - it’s the only way I’m able to do the import/export stuff at all.

Just confirmed it’s running out of memory (via dmesg) - it’s being killed by the Kubernetes memory controller - surely 512MB (the maximum capacity of the pod, burstable to 768MB) is more than sufficient for the script to run?

[29603219.885882] php invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null),  order=0, oom_score_adj=991
[29603219.896637] php cpuset=6f6e7c78c0696030ef55ce3f5ad052125f187f7a4f927e4094be1f9de3a44235 mems_allowed=0
[29603219.906366] CPU: 0 PID: 343425 Comm: php Not tainted 4.14.127+ #1
[29603220.360460] [343425]  1001 343425   124980    54339     230       3        0           991 php
[29603220.452305] Memory cgroup out of memory: Kill process 343425 (php) score 1417 or sacrifice child
[29603220.461504] Killed process 343425 (php) total-vm:499920kB, anon-rss:204096kB, file-rss:13260kB, shmem-rss:0kB
[29603220.513305] oom_reaper: reaped process 343425 (php), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Looks like I may have to try running this locally where I can provide more memory.

asmecher · June 3, 2020, 4:28am

Hi @dh_its,

Increasing the memory limit will likely solve the problem. We also recently added some alternatives to embedding file contents in the export XML that might be easier to work with and would definitely require less memory. See Extend XML native import/export plugin to support alternatives to embedding · Issue #5372 · pkp/pkp-lib · GitHub for details.

Regards,
Alec Smecher
Public Knowledge Project Team

dh_its · June 3, 2020, 8:22am

Hi Alec,

Will try that patch, it looks like it might work better for exporting.

Not sure why the first error occurs with content that was created in OJS 3.1 or later though (the missing issue and subsequent conversion failure)…

Thanks!
Daniel

asmecher · June 3, 2020, 5:45pm

Hi @dh_its,

I think the Missing child element(s) error is probably a data error. To track it down, find the code that validates the XML export snippet in lib/pkp/classes/xslt/XMLTypeDescription.inc.php:

if (!$xmlDom->schemaValidate($this->_validationSource)) return false;

Change it to…

if (!$xmlDom->schemaValidate($this->_validationSource)) {
    $errors = libxml_get_errors();
    error_log(print_r($errors,true));
    return false;
}

This should capture XML validation errors in the log. (We could certainly be doing a better job of error handling, rather than requiring this kind of intervention, and are gradually improving our practices!)

More broadly, though, are you looking to move content from one OJS installation to another? If so, is it an installation that contains other journals already, or would it be a new one with just the one journal? Or are you moving content to another system entirely?

Regards,
Alec Smecher
Public Knowledge Project Team

dh_its · June 4, 2020, 12:34am

Will do that then, I’ll let you know what the results are.

We are looking to move content from an existing multi-journal installation to a new installation that’s standalone, so it will be OJS to OJS.

Thanks,
Daniel

asmecher · June 4, 2020, 2:12am

Hi @dh_its,

In that case, here’s another way to do it that’ll preserve a lot more information:

Move the entire installation over to a new server (or new location on the same server) – a copy of the installation directory, database, files_dir, and with config.inc.php set up to point to the new database and files_dir
In the Administration interface, delete all the journals you don’t want to include in the new installation.

Regards,
Alec Smecher
Public Knowledge Project Team