How to import large xml files in OJS

Software Support
1

Hi,

I’m unable to import large xml files (50MB, 256MB and 351 MB)

I tried after changing post max size, max upload size and execution time in php.ini.

if i try to import from back end(tools folder) it is importing half of the articles and getting terminated automatically as Killed.

Kindly help me in importing large xml files

2

Hi! What version of OJS is it?

3

Hi ,

Thanks for the reply

I’m using OJS 2.4.2

4

If something outside of OJS is terminating the process, it is probably some form of memory manager running on your server. Check with your system administrator or hosting provider to see what the largest allowed memory size is for your processes. They may be able to increase this for you temporarily or permanently. You can also try splitting your XML imports into smaller parts. For example, you might export and import just one issue at a time.

5

Hi,

Thanks for the reply

Each issue size is of 250Mb/40Mb/351Mb

And i have kept all issues to be imported in external hard disk drive (250 GB is free) and our application is hosted on ubuntu and it has 6.5Gb free space

i want to transfer issues from one ojs instance to another(each instance present in different servers)
Any other ways to import?

6

Using the Native XML export/import is the best way to accomplish this transfer.

You can have all the disk space you need, but you will still be limited by whatever is regulating your memory usage.

Did you check with the system administrator or hosting provider about temporarily or permanently increasing your memory limits?

7

Still the same problem.

our system admin told us to run the process in background, even after that still its getting terminated after importing few articles

kindly help

8

If you are running the process in the background, I presume you are running the import from the command line. Can you share the command line you are using, and the exact output from the command, including the message about the process being “Killed”?

9

Please find the command below

php importExport.php NativeImportExportPlugin import 6155.xml journalPath editor &

10

How about the output of the command or logs, especially the message about the process being “Killed”?

11

Oct 14 06:38:42 ojs kernel: [11527786.609523] Out of memory: Kill process 46738 (php) score 761 or sacrifice child
Oct 14 06:38:42 ojs kernel: [11527786.619933] Killed process 46738 (php) total-vm:11510476kB, anon-rss:11245172kB, file-rss:0kB

12

You are exceeding your available system memory (RAM). You will need to cut down the size of the imports by breaking them into smaller chunks, or you will need to add memory to the machine instance. There may be another process running which is consuming a large amount of memory. Try running top or a similar command to get an idea of what your baseline memory usage is, and what is available for PHP.