I have a peculiar situation where I have an old version of a journal (on 2.4.x.x) running on Server A, with lots of issues and articles (about 1000) and a newer version of the same journal (on 3.3.x.x) running on Server B. The newer version is missing some (say 50) issues from the older version because they were imported after the newer version moved server.
Also all these articles are extremely data-heavy with lots of GB in .pdf files.
I can’t simply update the old journal to a new version, copy the database and run it instead of the new one, since the update is rather time consuming. Also there has been work done in the new journal, which I do not want to overwrite.
I would like to export the missing issues from the old journal to the new one in the most time efficient fashion.
The first thing I tried is exporting the Issues in 2.4 via native XML Plugin and importing them onto 3.3. However this approach is rather clumsy. ~5-10 issues export to 1GB of xml code, which takes ages to upload and dowload. Also while testing I have found out that the xml data can cause errors, or simply will not import into the 3.3 system.
Is there a recommended process for this kind of problem? Any way I can maybe split the /data/ folder from the XML Export, manually copy the data and then have the metadata added via XML?
It does not seem possible to combine the old version with the new one.
I know it is not possible to export via xml too.
What can you do? (in my opinion)
First you need to update OJS 2.4 to 3.3.0.8.
Then you can run work for combination if needed.
You said “I can’t simply update the old journal to a new version”. What kind of problem are you having?
I think server compatibility issues are a problem when updating, since the server is quite old. I can try though. Also updating from 2.4 to 3.x is like navigating a maze of errors, that I hoped to avoid.
There’s no possibility of separating the pdf files from the xml then during export?
Yes, you cannot export from 2.4 and import into 3.3. The XML schemas are not compatible. You will first need to upgrade the OJS 2.4 installation to OJS 3.3 and then perform your export. It is however not possible to upgrade directly from 2.4 to 3.3, so you will need to go to 3.2 first.
@kerimsarigul 's suggestion a local upgrade is probably the approach I would take as well.
Yes, essentially you’d set up an entire OJS installation on your own PC. That would mean running a web server and a database server, as well as PHP. You’d do this by exporting your database out of the server database, getting a copy of your OJS install and the files_dir directory, and then get all of that running on your PC. Then you’d upgrade it to 2.4 → 3.2 → 3.3 and then you’d be able to export your content and import it into your server 3.3 install.
It’s a lot of work but it’s really the only way unless you can upgrade your server 2.4 OJS installation.
Thanks to a helpful colleague I was able to update the server to Debian 10 and PHP 7.3.
I have since tried to update the journal from 2.4.8 to 3.1.2-4 (which has been a “safe” stepping stone for updates to 3.2+ in the past), but I have encountered a rather odd error which I haven’t seen before:
[Fri Feb 04 11:50:53.678360 2022] [:error] [pid 21840] [client 132.199.243.26:51798] PHP Warning: Constants may only evaluate to scalar values in /var/www/ojs-3.1.2-4/lib/pkp/classes/db/DBConnection.inc.php on line 17
[Fri Feb 04 11:50:53.680330 2022] [:error] [pid 21840] [client 132.199.243.26:51798] PHP Parse error: syntax error, unexpected '?', expecting variable (T_VARIABLE) in /var/www/ojs-3.1.2-4/lib/pkp/classes/notification/PKPNotificationOperationManager.inc.php on line 373
this error occurs, when I try to run the online installer. The Page stays blank and this error is found in the log.
Any idea what may cause this?
I have both tried defining the database as “mysql” and “mysqli” in the config.inc.php, since that has been a source for errors in the past, but I get the same error either way.
That PHP Parse Error is due to you attempting to use PHP 5.6, instead of PHP 7. You might have installed PHP 7.3 generally but please make sure Apache is also loading the 7.3 PHP module as well.
Thank you very much.
I managed to get it to work in the end. Now I can Export Issues to XML, download them, and upload them to the new server.
But, the Import isn’t working.
I worked around one error as shown in this thread:
When trying to import, I’m getting these two Errors:
PHP Warning: DOMDocument::loadXML(): Memory allocation failed : growing input buffer in /var/www/ojs_albertina/plugins/importexport/native/NativeImportExportPlugin.inc.php on line 379
PHP Warning: DOMDocument::loadXML(): Memory allocation failed : growing input buffer in /var/www/ojs_albertina/lib/pkp/classes/xslt/XMLTypeDescription.inc.php on line 120
The script seems to run through, but nothing is imported, at least I can’t find anything in the journal.
I have made the import files as small as plausible, but they’re still ~1GB per File.
max_memory and max_upload in php.ini are both set to 4G, so that should not be a problem.
You’ll need to raise the allowed memory limit for PHP. How you do this really depends on your server configuration but it is often done either via an .htaccess file or a .user.ini file in the web root. Here’s a quick link to the .user.ini version that you could try first.
I ran it again and I still get the same errors, as well as a new PHP Fatal Error:
PHP Fatal error: Uncaught PDOException: SQLSTATE[HY000]: General error: 1114 The table 'controlled_vocabs' is full in /var/www/ojs/lib/pkp/lib/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php:115
Stack trace:
#0 /var/www/ojs/lib/pkp/lib/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php(115): PDOStatement->execute(NULL)
#1 /var/www/ojs/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Database/Connection.php(489): Doctrine\DBAL\Driver\PDOStatement->execute()
#2 /var/www/ojs/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Database/Connection.php(664): Illuminate\Database\Connection->Illuminate\Database\{closure}('INSERT INTO con...', Array)
#3 /var/www/ojs/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Database/Connection.php(631): Illuminate\Database\Connection->runQueryCallback('INSERT INTO con...', Array, Object(Closure))
#4 /var/www/ojs/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Database/Connection.php(496): Illuminate in /var/www/ojs/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Database/Connection.php on line 671
I’ve never read anything about a Illuminate Database Connection before, so I’m pretty stumped right now.
Some Quick googling suggests that the size of the table might be too big, or the disk space might have run out, or the Database settings don’t allow filesizes that big. I’m gonna check that.
Well, that’s progress at least - Illuminate is just the database package that OJS uses in 3.3.
As for the MySQL error, if a df -h command on the command line doesn’t indicate that a disk partition is full, this is probably a MySQL setting that needs to be raised. OJS 3.3 requires InnoDB database tables which means that there may be a setting around the the ibdata table size maximum that needs to be raised especially if your XML files are as large as they seem to be.
Progress has been made. Indeed it was the diskspace being the problem.
Now the import seems to run through. I got the following warnings, which are no big problem:
Warnings encountered:
1.Issue
- Existing issue with id 91 matches the given issue identification "<issue_identification>
<title locale="de_DE">[...]</title>
</issue_identification>". This issue will not be modified, but articles will be added.
2.Issue
- Existing issue with id 92 matches the given issue identification "<issue_identification>
<volume>26</volume>
<year>1966</year>
<title locale="de_DE">[...]</title>
</issue_identification>". This issue will not be modified, but articles will be added.
3.Issue
- Existing issue with id 93 matches the given issue identification "<issue_identification>
<volume>17</volume>
<year>1924</year>
<title locale="de_DE">[...]</title>
</issue_identification>". This issue will not be modified, but articles will be added.
I can find the imported issues in the frontend, when browsing the Archive. I can’t find the issues in the backend though, which means I cannot alter them in any way. Is this the expected behaviour?
Thanks.
I found the three Issues which had warnings, which as I read them mean that they were there before. I can’t find the other issues which should have been imported at all though. (neither in the frontend, nor backend)
Is there a possibility to debug the import process, or make it more verbose? I’m struggling to find out where the error may be.
There’s not really a great way to debug the import process other than adding a whole lot of error_log() calls in the code there; the Filter code in OJS is heavily obfuscated and difficult to read, unfortunately.
It would be very weird if issues are appearing on the front end and they work completely, including galley viewing, but they do not appear in the back end in either the back issues area or the Archived submissions tab. The usual sign of a broken import is “Invalid JSON or AJAX returned” or other error popups indicating broken tabs. If all of the tabs work, but the submissions aren’t there, and yet they appear on the front end, that’s a first for me. Have you tried clearing your Caches in the administration area?
Are they in the database? Can you possibly compare records between one that is appearing and one that is not? Issues are assigned to submissions in the publication_settings table via a setting_name of issueId.
I suppose one thing I should mention. After your last failed import (when you ran out of disk space), did you restore the OJS installation you were importing into from a backup? That failed import could have left issue and submission information in a partially imported state. We recommend making a backup of the installation before you perform an import in case the import fails, as the import process will not roll back if something wrong happens.
Hi Jason,
I have in fact not imported into a “fresh” installation. Backups are daily, so I have Backups to go to, but People have worked on the system in the meantime.
I wasn’t aware that the import progress could destroy the database.
I will try another import file, if that yields the same results I will have to backup and contact the people working on the platform.
Is there another way to clean partly imported information?
Suggestion: please add a “Only import into fresh backups, incomplete imports may corrupt your database” warning to the admin guide, as this is the first time I hear of that fact and it’s mighty inconvenient.
Point taken about the documentation addition, but we do always recommend - and include in our upgrading documentation that ships with OJS on the very first line - that you should create a full backup before beginning any work on the database or your installation. In your case, you ran out of disk space in the middle of a large import, and there’s really nothing MySQL or OJS could have done to safely recover from that, even with the best error handling.
The only other way to clean partially imported information is to go through your database one submission at a time and remove the records in all of the various tables pertaining to those submissions, although this is quite error prone and - in OJS 3.3, with foreign keys now used in the database - difficult to do easily since MySQL will prevent you from deleting records in tables that have constraints pointing to other tables unless you disregard those foreign keys. Strongly recommend a restore here. When I use this plugin I import into a staging server first, a completely separate second installation, so I know that the import is going to work cleanly before I attempt it in on a live site.