Testing XML imports for v3.2.1-1

cbirmingham · July 31, 2020, 8:31pm

Having recently installed OJS v3.2.1-4, I’ve moved on to testing XML imports.

Logging into the web interface, I’ve noted that all of the boxes for import/export plugins are checked. Based on what I read in the documentation here: , I downloaded the sample XML file for issue metadata.

When I attempt to upload that sample XML file via ‘Tools > Import/Export > Native XML Plugin > [ Upload File ]’, I get a brief flash of a progress bar, then an error pop-up that says “The file could not be uploaded or revised.”

Based on that whole experience, my initial questions are:

What other component or setting do I need to address before this trial XML import will succeed?
Is there any way to set/adjust file size limits for XML import files? I noticed that there’s a way to adjust the ‘public’ directory for users, so I’m wondering if there are ways to set/adjust similar limits for actual files.

asmecher · August 1, 2020, 2:26am

Hi @cbirmingham,

A few things to check?

Are you importing XML documents that were exported from the same version of OJS? If not, some adaptation may be necessary.
Does the import XML validate against the schema? You can check this with any XML validator. (I use xmllint, which is available for most platforms.)
Are your file permissions set correctly in the files_dir (configured in config.inc.php)? If you installed using the command line PHP tool, but are importing using the web-based tool, the file permission requirements may be different for each case.
Check your PHP error log to see if there are any details there.

OJS doesn’t limit upload file sizes, but your PHP and/or web server environment will.

Regards,
Alec Smecher
Public Knowledge Project Team

cbirmingham · August 3, 2020, 3:53pm

Hi @asmecher,

For the first suggested check, I figured that the sample XML files were for the latest version of OJS. If that isn’t the case, can you provide links to sample files for v3.2.1-1?

For the second suggested check, I copied this XML file to my local system as 'test-native-xml-issue.xml and this XSD file file to my local system as ‘test-native-xml.xsd’. I then tried to validate the XML file with th following command: xmllint --schema test-native-xml.xsd test-native-xml-issue.xml.

I got the following error:
warning: failed to load external entity "../../../lib/pkp/plugins/importexport/native/pkp-native.xsd" test-native-xml.xsd:18: element include: Schemas parser error : Element '{http://www.w3.org/2001/XMLSchema}include': Failed to load the document '../../../lib/pkp/plugins/importexport/native/pkp-native.xsd' for inclusion. WXS schema test-native-xml.xsd failed to compile

Is there a different XSD file I should be using to validate that sample XML file?

For your third and fourth suggestions, I’ll check around on my test OJS system and see what I can find regarding the current settings and what’s in the error log.

cbirmingham · August 4, 2020, 9:14pm

Hi @asmecher,

For your third suggested check, I blew the instance away and did a fresh rebuild. in config.inc.php, the original umask value for my /opt/ojs directory was ‘022’, which I believe corresponds to a chmod 644 permission set. Since I manually set that directory’s permissions to 664, I adjusted the umask to 002 to match. I still got a "The file could not be uploaded or revised.” error. I rebuilt the system again, setting the umask value to 000, which should mean that OJS sees it as wide open, but got the same error. This brings me to your fourth suggested check.

For your fourth suggested check, I looked at the apache error log (/var/log/apache2/error.log) right after building the system (with umask 000) and saw only standard stop/start messages. Then I created a journal through the web interface (which appeared to work just fine) and checked again. There was a significant amount of warnings/errors in the log just from that journal creation. From there, I then tried to import an XML file and got a few more errors from that.

Here is just the piece from the XML import attempt:

80a81,85

[Tue Aug 04 16:19:30.093335 2020] [php7:warn] [pid 22825] [client 73.48.124.76:37999] PHP Warning: mkdir(): Permission denied in /var/www/html/ojs
/lib/pkp/classes/file/FileManager.inc.php on line 311, referer: http://journals.lib.fsu.edu/ojs/index.php/sandbox/management/importexport/plugin/Nati
veImportExportPlugin
[Tue Aug 04 16:19:30.093376 2020] [php7:notice] [pid 22825] [client 73.48.124.76:37999] PHP Notice: tempnam(): file created in the system’s tempor
ary directory in /var/www/html/ojs/lib/pkp/classes/file/TemporaryFileManager.inc.php on line 90, referer: http://journals.lib.fsu.edu/ojs/index.php/s
andbox/management/importexport/plugin/NativeImportExportPlugin
[Tue Aug 04 16:19:30.093433 2020] [php7:warn] [pid 22825] [client 73.48.124.76:37999] PHP Warning: mkdir(): Permission denied in /var/www/html/ojs
/lib/pkp/classes/file/FileManager.inc.php on line 311, referer: http://journals.lib.fsu.edu/ojs/index.php/sandbox/management/importexport/plugin/Nati
veImportExportPlugin
[Tue Aug 04 16:19:30.093463 2020] [php7:warn] [pid 22825] [client 73.48.124.76:37999] PHP Warning: move_uploaded_file(/opt/ojs/temp/xmlQASR9i): fa
iled to open stream: Permission denied in /var/www/html/ojs/lib/pkp/classes/file/FileManager.inc.php on line 141, referer: http://journals.lib.fsu.ed
u/ojs/index.php/sandbox/management/importexport/plugin/NativeImportExportPlugin
[Tue Aug 04 16:19:30.093471 2020] [php7:warn] [pid 22825] [client 73.48.124.76:37999] PHP Warning: move_uploaded_file(): Unable to move ‘/tmp/phpd
Dzv5L’ to ‘/opt/ojs/temp/xmlQASR9i’ in /var/www/html/ojs/lib/pkp/classes/file/FileManager.inc.php on line 141, referer: http://journals.lib.fsu.edu/o
js/index.php/sandbox/management/importexport/plugin/NativeImportExportPlugin

It might be possible to diagnose what’s going on with the XML imports withough addressing all of the errors that cam from creating a journal, but I’m not sure how that would work. If you’d like to look through my error log files, what would be the best way to send them to you?

Thanks,
Chuck Birmingham II
FSU Libraries

asmecher · August 4, 2020, 9:27pm

Hi @cbirmingham,

Does your server have SELinux extensions enabled? That might explain the trouble you’re having with getting umask/mode/owner combinations to work as OJS expects.

As for the message from the command xmllint --schema test-native-xml.xsd test-native-xml-issue.xml:

warning: failed to load external entity "../../../lib/pkp/plugins/importexport/native/pkp-native.xsd" test-native-xml.xsd:18: element include: Schemas parser error : Element '{http://www.w3.org/2001/XMLSchema}include': Failed to load the document '../../../lib/pkp/plugins/importexport/native/pkp-native.xsd' for inclusion. WXS schema test-native-xml.xsd failed to compile

This is appearing because the XSD refers to other XSD files that you haven’t downloaded. If you’re in the OJS directory, you can just use…

xmllint --noout --schema plugins/importexport/native/native.xsd /path/to/import-document.xml

…and you should get:

/path/to/import-document.xml validates

The sample.xml document included with OJS was recently noted as being out of date. See [OJS] ImportExport XML sample file deprecated · Issue #6121 · pkp/pkp-lib · GitHub for details. You can get a sample XML document for OJS 3.2.1-1 from ojs/export-issues.xml at 3_2_1-1 · pkp/ojs · GitHub.

Regards,
Alec Smecher
Public Knowledge Project Team

cbirmingham · August 5, 2020, 7:11pm

Hi, @asmecher,

Regarding your SELinux suggestion, I did another fresh install (setting the umask value to 002) and ran some basic SELinux commands. This got me thinking that maybe Ubuntu’s UFW might have been enabled, so I checked for that as well. Here’s what I saw:
selinux-ufw-status

From there, I went only once to the main /ojs page. Here’s what I saw in /var/log/apache2/error.log:
single-access-errors

So it looks like the apache/php errors I am seeing aren’t related to creating a journal, but just to hitting the site. If SELinux isn’t even installed, and UFW is inactive, I’m not sure what the permissions issue(s) could still be.

Regarding your input on the XML/XSD part, thanks for letting me know those details and where to find the current/correct sample XML file. Downloading the proper file, and running the command you included, I got this:
xmllint-attempt

Could this also be indicative of some kind of permissions issue?

asmecher · August 5, 2020, 7:50pm

Hi @cbirmingham,

On file permissions…

You can isolate OJS out of the mix by creating a script…

<?php
ini_set('display_errors', 1);
error_reporting(E_ALL);
file_put_contents('/var/www/html/ojs/cache/test-file.php', 'test');

Call it e.g. test.php, put it in your web root, and invoke it via your web browser. (Note: don’t invoke it using command-line php – that won’t be a useful test.) I expect you’ll see the same permission errors.

If you do, you’ll have to work with your server config, file permissions, etc. to figure out what your server’s particular requirements are to allow PHP scripts to write files there. SELinux was only a guess, and it does look like it’s not installed on your system.

On import/export XML…

Have a look at the file you saved using wget and you’ll see it’s HTML-based pretty-printing, not the XML document you expect. You can use the Raw link at the URL above to get the raw XML file.

Regards,
Alec Smecher
Public Knowledge Project Team

cbirmingham · August 10, 2020, 4:44pm

Hi @asmecher,

On file permissions…
A coworker of mine suggested that I try adjusting permission on the upload directory to 775. It turns out that making that adjustment fixed the problem. I can now install via the web interface as well as the command line. and my Apache log file is warning/error-free.

On import-export XML…
Indeed, downloading the raw-format version took care of this issue.

I’d say that the issues here are sorted out.

Thanks again,
Chuck Birmingham II
FSU Libraries