Missing data in Usage statistics report (OJS 3.1.1.2)

Hello, I have run the [Usage statistics report] for all our journals and received very detailed information (so I am pleased with it…). However, in one of our journals data stop with 202010, while all remaining reports return data up to 202101. Strange is that this is our older journals and I cannot figure out why this happens. I do not handle any codes, so use a standard installation.
Does anyone have an explanation about this problem?
Thank you,
Lucia

Do some of your journals have their own base_url configuration? For example, do some of the journals have a url like myjournal.com and others have a URL like myojs.com/index.php/otherjournal?

Hello Nate and thank you for your reply. I have checked and all our journals have this URL base, the same for all 5 journals on the installation:

AboutSciencejournal/index

Going through the reports I noticed that
journal 1 (gcnd): month starts at 201910 and ends at 202002 for a total of 4992 lines
journal 2 (ao): month starts at 201809 and end at 202010 for a total of 4886 lines

Other 3 journals are just fine and with very granular information. I am wondering if there is a “line limit”?
As you can image this can be a problem in my Y-end reporting activities.
Thank you for your time!
Lucia

I’m pretty sure there is not a line limit. I’m not sure what would cause stats to be missing from some journals after a certain date. To my knowledge, there’s no way to disable usage stats on a journal-by-journal basis.

The first thing I’m look into is whether or not log entries are getting recorded correctly for those journals. You can see log entries before they are processed in your files_dir, under the usageStats/usageEventLogs directory. Open up the files there to see if you can spot entries for the journals in question.

Hello Nate, I cannot find these event logs (I am not actually a programmer…). I have raised a ticket with our provider and hopefully they can look into it.
Will get back to you if I received further information and data.
Thanks for now…
Lucia

Hi everyone,

I am facing a similar situation. On article stats graphic, there is a period without data, zero metrics, but no website availability issue.

There is several journals in OJS installatiion, all of then with the same graphic.

On files_dir/usageStats there is:

  • archive: 65 files, from usage_events_20220412.log to usage_events_20240702.log, with no regular data, eg: in sequence usage_events_20230803.log, usage_events_20230804.log, usage_events_20240506.log and usage_events_20240702.log (latest files).
  • processing: 1 file from 2023 usage_events_20230919.log
  • reject: 17 files, from usage_events_20220722.log to usage_events_20230423.log
  • stage: 730 files, from usage_events_20220522.log to usage_events_20240722.log
  • usageEventLogs: 2 files - usage_events_20240723.log and usage_events_20240724.log

It seems stage files is not beeing proccessed. I do not know where to look for some issue.

error_log file has only:

  • Several (daily entries) PHP Warning: Illegal string offset 'pt_BR' in /public_html/lib/pkp/classes/core/DataObject.inc.php on line 133
  • Several (daily entries) PHP Warning: Cannot assign an empty string to a string offset in /public_html/lib/pkp/classes/core/DataObject.inc.php on line 133

I do not believe these warnings has relation with usageStats issue, but… Any idea @NateWr ?

@Isteele, have you solved your issue?

OJS version: 33014
PHP version: 7.4

Hi @geniusdesign

Hmmm… It seems there is a problem with your log file processing. Maybe you can take a look at the scheduled task log files – if you would see any error or hint there. You will find the relevant scheduled task log files in your files_dir/scheduledTaskLogs/ folder. Take maybe a look into the most recent log file with the name UsageStatsLoader-…-date.log, but maybe also for the same dates as the usage stats log files that are in the reject folder.
Else, double check that your file permissions are correct (e.g. that the web user can move/copy the files from one folder to another).
Also, because there is a huge amount of the unprocessed log files, maybe to backup them somewhere else first and then try re-processing them one by one, watching the appropriate scheduled task log file for errors/warnings. Then solving the problems and re-processing them further.

Best,
Bozana

Hi @bozana !

Thank you very much for your reply. Actually, I have found the solution (hope so) this morning. Reading carefully the documentation (https://docs.pkp.sfu.ca/admin-guide/3.3/en/statistics#processing-log-files), I realized that I was using an incorrect command through ssh.

I was using only php tools/runScheduledTasks.php as command and nothing happened. When I used “Reload scheduled tasks” on Acron plugin (via OJS interface) nothing happened too, so I thought there was any bigger issue.

After reading docs, I moved the file stucked on processing folder to stage folder and ran the correct command php tools/runScheduledTasks.php plugins/generic/usageStats/scheduledTasks.xml. I also moved all 17 files on reject folder to stage folder before run the correct command.

Here is the documentation instruction about it:

Note that for any process you choose, you can move files into the stage folder anytime, even while the scheduled task is running. You can also move any number of files inside the stage directory. What determines the period of time that you will be moving files into the stage directory is mainly your necessity for updated statistics.

After doing it, the command on ssh is still runnin (over than 2 hours from this reply) and the number of files on stage folder is decreasing. It was with 748 total log files and now, about 2 hours later, it has 712 files.

At this point, reject folder has 1 file (usage_events_20230423.log) only and archive folder has 101 files. I believe it is everything runing well.

When it is over, I will move reject files to stage folder and run again.

Thank you very much for your time and hope it helps who need.

My article stats graph is more beautiful already =)

\o/

Hi @geniusdesign

Great! Glad to hear that! :-)))

Dear all,

I probably have the same problem as @geniusdesign (my OJS version is 3.3.0.19). Therefore I will not open a new thread but continue this one. My usage_event.logs are not processed correctly either, which leads to big gaps in our statistics. We moved to a new web server at the beginning of August, but the problems only started to appear at the beginning of October. It has become even worse in recent months.

There are a total of 238 files in stage directory that are not being processed. Some files are quite large at 6 megabytes. Is this normal? The average is around 500 kilobytes. There is a file in processing that probably cannot be processed any further and is clogging the process. There are two in reject. There are 2500 files in archive.

The error logs in the scheduledTaskLogs directory are always the same. Here is the UsageStatsLoader-679658a442c2d-20250126.log (note: I have translated the error messages from German into English so maybe the wording differs from the original English error messages):

[2025-01-26 16:45:40] https://zeitschrift-suburban.de/sys
[2025-01-26 16:45:40] [Note] Scheduled task has been started.
[2025-01-26 16:45:40] [Error] The directory /var/www/vhosts/hosting204046.ae86a.netcup.net/ojs-files/usageStats/processing is not empty. This may indicate a previously failed or currently running process. This file will be automatically reprocessed if you are also using scheduledTasksAutoStage.xml; otherwise you will need to manually move all orphaned files from the processing directory back to the staging directory.

Like @geniusdesign, I moved the files from the processing and rejected into the stage directory and executed the command via ssh:

php tools/runScheduledTasks.php plugins/generic/usageStats/scheduledTasks.xml

However, the console is unblocked immediately after the command is executed so that I can enter new commands. I am therefore unsure whether the process is started at all. Is there any way I can check this (OJS 3.3.0.19)? Even two hours after executing the command, there is no file in the processing directory. The latest UsageStatsLoader.log file is from yesterday. Is this evidence that the process is not being triggered?