Views and download of articles are not counted anymore since upgrade OJS 3.1.2

Software Support
1

Hi,

We just noticed that since we upgraded to OJS 3.1.2, views and download of articles are not counted anymore.
usage_events_*.log files stay in directory usageStats/usageEventLogs and they are no more processed.
They don’t exist in metrics table in DB.

scheduledTaskLogs/Usagestatisticsfileloadertask* don’t exist

This SQL command gives the following result:
SELECT * FROM scheduled_tasks where class_name=‘plugins.generic.usageStats.UsageStatsLoader’
plugins.generic.usageStats.UsageStatsLoader 2019-04-30 07:33:19
So UsageStatsLoader seems to be executed

We don’t change our config file:
scheduled_tasks = On

Acron plugin is activated

It’s the first time it’s happen even when we upgraded from OJS2 to OJS3

I had a look to this statistic documentation:

I tried to execute manually this command:
php tools/runScheduledTasks.php plugins/generic/usageStats/scheduledTasks.xml

But I got this error:
PHP Fatal error: Cannot redeclare geoip_country_code_by_name_v6() in /ojs_home/plugins/generic/usageStats/lib/geoIp/geoip.inc on line 430

I don’t know what to do and I will need a little help for this.

Thanks in advance for your answer.
Best regards.
Helene

2

What version did you upgrade from, and what method did you use to upgrade?

It sounds like you have a duplicate copy of the geoIp library installed in your system.

Prior to 3.1.2, this library was installed in lib/pkp/plugins/generic/usageStats/lib/geoIp. With 3.1.2, this library was moved out of the shared library and directly into the usageStats plugin.

3

Hi @ctgraham,

I installed the OJS 3.1.2 released by downloading the whole archive from the official website.
The previous release installed was OJS 3.1.1.4
usageStats is not present in “lib/pkp/plugins/generic/” directory and is only present in “plugins/generic/” directory.
I searched for other occurence of geoip_country_code_by_name_v6 in OJS home directory but I didn’t find any.

I never tested to execute manually this command in previous release:
php tools/runScheduledTasks.php plugins/generic/usageStats/scheduledTasks.xml

Is it the good way to do execute it ?
What will it do exactly?
Does this relate to the fact that usage_events_*.log files stay in directory usageStats/usageEventLogs and they are no more processed ?

Thanks again for your help.
Kind regards.
Helene

4

HI @ctgraham and @hcl,

I have a similar or the same problem: last week I migrated from v3.1.1.4 to v3.1.2 and the configuration of the static plugin looks fine.

But when generating a report it doesn’t contain any data besides column titles.

The logging seems to work, as I have a large number of daily generated log files in the folder:

/usageStats/archive

It seems, the logged data can’t be interpreted by OJS. What can I do to solve this problem?

Thanks a lot for advice,
Anna

5

OJS stores log files initially in the “usageEventLogs” directory. The log files must be moved to a “stage” directory before being processed (in the “processing” directory) into either the “archive” directory (for success), or the “reject” directory (for failure).

https://pkp.sfu.ca/wiki/index.php?title=PKP_Statistics_Framework#Processes

Running the command php tools/runScheduledTasks.php plugins/generic/usageStats/scheduledTasks.xml will process any OJS log files which have been moved to the “stage” directory.

You probably instead want to run the command for autostaging:
php tools/runScheduledTasks.php plugins/generic/usageStats/scheduledTasksAutoStage.xml
This will move the logs into the “stage” directory automatically for processing.

6

If logs have made it to the “archive” directory, OJS thinks they have been processed successfully.

You can find additional details in the scheduledTaskLogs. This may indicate lines that have been skipped within the successful processing.

You will want to check your scheduledTaskLogs and check the metrics table in the database in order to identify whether the detail from the log files has been successfully extracted into metrics.

7

Hi @ctgraham,

It seems that no scheduledTaskLogs folder exists. The permissions and users of the files folder are correct (0755), so this can’t be the problem.

Here a short excerpt from the metrics table:

| usage_events_20160512.log | 521 | 8 | 4997 | 4245 | 20160512 | 201605 | 1 | NULL | NULL | NULL | ojs::counter | 1 | NULL | NULL | NULL | 4245 |
| usage_events_20160512.log | 521 | 8 | 4988 | 4247 | 20160512 | 201605 | 1 | NULL | NULL | NULL | ojs::counter | 2 | NULL | NULL | NULL | 4247 |
| usage_events_20160512.log | 521 | 8 | 4987 | 4249 | 20160512 | 201605 | 1 | NULL | NULL | NULL | ojs::counter | 1 | NULL | NULL | NULL | 4249 |
| usage_events_20160512.log | 521 | 8 | 4981 | 4252 | 20160512 | 201605 | 1 | NULL | NULL | NULL | ojs::counter | 1 | NULL | NULL | NULL | 4252 |
| usage_events_20160512.log | 521 | 8 | 5006 | 4253 | 20160512 | 201605 | 1 | NULL | NULL | NULL | ojs::counter | 1 | NULL | NULL | NULL | 4253 |
| usage_events_20160512.log | 521 | 8 | 5003 | 4255 | 20160512 | 201605 | 1 | NULL | NULL | NULL | ojs::counter | 1 | NULL | NULL | NULL | 4255 |

Should I run the command

php tools/runScheduledTasks.php plugins/generic/usageStats/scheduledTasksAutoStage.xml

In the apache log files I can’t find any error logs which indicate problems with the statistic plugin.

thanks for your support,
Anna

8

Hi @asmecher,

The last log file in “/var/files/usageStats/archive” directory is from the day before the upgrade to OJS 3.1.2
The log files in “/var/files/usageStats/usageEventLogs” directory stayed in this directory since the day of the upgrade to OJS 3.1.2

There are not log files in “/var/files/usageStats/processing”, “/var/files/usageStats/stage”, “/var/files/usageStats/reject” directories when I looked

When I run the command for autostaging:
php tools/runScheduledTasks.php plugins/generic/usageStats/scheduledTasksAutoStage.xml

Nothing happens!

I noticed when I removed the entry in the DB table scheduled_tasks with the class_name = plugins.generic.usageStats.UsageStatsLoader
And then I ran again the command for autostaging, I have the same error when I run the usageStats/scheduledTasks.xml command:
PHP Fatal error: Cannot redeclare geoip_country_code_by_name_v6() in /ojs_home/plugins/generic/usageStats/lib/geoIp/geoip.inc on line 430

I don’t know why I have got this error and if it’s the reason why logs are not moved into the “stage” directory automatically for processing.

I also noticed the time registered in scheduled_tasks for the execution of plugins.generic.usageStats.UsageStatsLoader is not the same that the time server (2 hours difference). I don’t know if it’s relevant. It was maybe the case before.

I don’t know what is wrong, where I have to looking for, and what can I do as tests.

Have you any idea of what the problem could be?

Thanks again for your precious help.

Kind regards.
Helene

9

The scheduledTasksLogs directory should exist directly under your files_dir, parallel to the usageStats directory, which contains archive, processing, reject, etc.

The lines from the metrics table reference metrics ingested from event logs from 2016-05-12. Do you see recent entries as well?

You generally want to choose to either run the “tools/runScheduledTasks.php” command in cron, or use the acron plugin, which will automatically run the “scheduledTasksAutoStage” task. Do you have the acron plugin enabled?

10

The usage statistics processing is rate-limited and will not immediately run again unless the time indicated in the scheduled_tasks table has expired. If you clear the entry from the scheduled_tasks table and the command runs again, this is due to the rate limiting.

This is probably related to the value of the PHP date.timezone configuration. It should not cause a failure of processing the usage statistics, but you should set the value in your php.ini (or set time_zone in config.inc.php).

The PHP Fatal error is a concern. Since you have searched for geoip_country_code_by_name_v6 within the OJS codebase and only found the one occurrence, one possibility is that the geoip.inc file is being included twice somehow. If this were the case, however, it should error on the first function in that file (geoip_load_shared_mem), not geoip_country_code_by_name_v6.

11

Hi @asmecher,

Acron plugin is activated.

The scheduledTasksLogs directory do exists under my files_dir, parallel to the usageStats directory.
The last Usagestatisticsfileloadertask*.log is the day of upgrading to OJS 3.1.2. It’s written inside that the “/var/files/usageStats/processing/usage_events_yyyymmdd.log” was processed and archived.
yyyymmdd is the date of the day before upgrading to OJS 3.1.2

The last records in metrics table are with the last usage_events_yyyymmdd.log file stored in archive directory with the date of the day before upgrading to OJS 3.1.2

Processing usage events logs stopped when I upgraded to OJS 3.1.2 but I don’t know why.

Best regards.
Helene

12

In the Acron plugin, can you try running the “reload scheduled tasks” action? If Acron is enabled, I would expect it to be automatically running the Usage Statistics File Loader task.

13

Hello again @asmecher,
I had already thought about it and I already did it but it didn’t change anything.
Helene

14

Hi @ctgraham,

this afternoon I manually generated the folder

scheduledTaskLogs

and it was immediately filled with serveral log files. The log file for the usage statistics contains several warnings, that the usage_events_log file contains an adress where the system can’t remove the base url. What does that mean?

[2019-05-09 11:25:21] [Warnung] Die Zeile 7914 aus der Datei /opt/ojs-mthz-files/usageStats/processing/usage_events_20190508.log enthält eine Adresse, aus der das System nicht den Basis-URL entfernen kann.

The metrics table doesn’t contain recents entries, although there are daily event log-files in the archive-folder since 2016 up to today.

The acron plugin is enabled. I just restarted it and the event log files was re-generated, but the reports and counter statistics are still empty:

empty_counter

Do you have an more idea what goes wrong? We need theses statistics on monday…

thanks, Anna

15

Does line 7914 of usage_events_20190508.log look like an OJS URL? If so, it probably means that OJS failed to process the line. This can happen if OJS can’t correctly determine the hostname and any path for the URL. For example, if the line reads:

127.0.0.1 - - "2019-03-25 19:22:53" http://dev.hostname.tld/ojs3/default/index 200 "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0"

But if your config.inc.php references a base_url for something other than http://dev.ojs.hostname.tld/ojs3, then OJS may be failing to process that particular line. If all lines reference this warning, none of the lines are actually processed.

If you can correct the base_url settings in config.inc.php you can re-process the archived log files by moving them back from the “archive” folder to the “stage” folder. Try it with one file first and (if successfully processed) move all of the files from the missing date range. Reprocessing the files will overwrite any existing rows in the metrics table which match that load_id.

16

Hi @ctgraham,

you are right: the base_url in our config.inc.php differes from the hostname in the usage_events-20190508.log-file! I corrected it and copied the log file again into the staging area.

How can I trigger a new processing of this file?

thanks, Anna

17

This will happen automatically on the next acron run (1 day from the last run, unless you modify the scheduled_tasks table), or based on a manual run from the command line (as described above).

18

hi @ctgraham,

it seems to work!! Thank you very very much for your support :smiley:

best regards, Anna

19

Hi @ctgraham,

I still have the same problems. I tried to looking for what happened when I execute the scheduledTasks commands by displaying some messages in the code step by step:

When I execute the command:
php tools/runScheduledTasks.php plugins/generic/usageStats/scheduledTasks.xml
I have this error:
PHP Fatal error: Cannot redeclare geoip_country_code_by_name_v6() in /ojs_home/plugins/generic/usageStats/lib/geoIp/geoip.inc on line 430

I found the error occurs when in file :
/ojs_home/plugins/generic/usageStats/UsageStatsLoader.inc.php
this command line is executed:
$geoLocationTool = $statsHelper->getGeoLocationTool();
I found that function “getGeoLocationTool()” is called in this file:
/ojs_home/lib/pkp/classes/statistics/PKPStatisticsHelper.inc.php
And the error occurs after this command line is executed:
$geoLocationTool = $plugin->getGeoLocationTool();

When I execute the command:
php tools/runScheduledTasks.php plugins/generic/usageStats/scheduledTasksAutoStage.xml
Nothing happens
I found that in the following file:
/ojs_home/lib/pkp/classes/cliTool/ScheduledTaskTool.inc.php
In function “parseTasks”
When “$frequency” variable is tested, it has got a value, and this value is tested with this function:
$canExecute = ScheduledTaskHelper::checkFrequency($className, $frequency);
The checkFrequency “function” is included in this file:
/ojs_home/lib/pkp/classes/scheduledTask/ScheduledTaskHelper.inc.php
“$frequency” variable is an array which not contains month or day. But it contains an hour value of “0” which is consider not valid:
[attributes] => Array([hour] => 0)
That’s the reason why, “$canExecute” variable is false and nothing is execute after

Do you have any idea why in the 1st command, the call of “plugin->getGeoLocationTool()” command generates the following error ?:
PHP Fatal error: Cannot redeclare geoip_country_code_by_name_v6() in /ojs_home/plugins/generic/usageStats/lib/geoIp/geoip.inc on line 430

And why in the 2nd command, the presence of the “hour” parameter with the value “0” in “$frequency” variable generates an error ?
Where does the “hour” parameter come from? And where can I change it ?

Thanks again for your help.
Kind regards.
Helene

20

The hour parameter’s 0 value comes from:

This should be checked against:

This checks to see if the current hour has passed the hour of the last runtime. Thus, the scheduledTasksAutoStage.xml can only be run once a day.

I’m not able to reproduce the problem of Cannot redeclare geoip_country_code_by_name_v6() in my install of 3.1.2 stable. I’m very curious about what is causing this. If you search for ‘geoip.inc’ in your OJS files, do you find any references other than:

$ grep -r 'geoip.inc' *
plugins/generic/usageStats/lib/geoIp/README:Just place the 'geoip.inc' file somewhere according to the 'include_path' directive of
plugins/generic/usageStats/lib/geoIp/README:include("geoip.inc");
plugins/generic/usageStats/lib/geoIp/benchmark.php:include("geoip.inc");
plugins/generic/usageStats/lib/geoIp/geoip.inc:/* geoip.inc
plugins/generic/usageStats/lib/geoIp/geoipcity.inc: *              Changed inclusion of geoip.inc file to require_once, so that
plugins/generic/usageStats/lib/geoIp/geoipcity.inc: *                  this library can be used in the same script as geoip.inc.
plugins/generic/usageStats/lib/geoIp/geoipcity.inc:require_once 'geoip.inc';

?