We just noticed that since we upgraded to OJS 3.1.2, views and download of articles are not counted anymore.
usage_events_*.log files stay in directory usageStats/usageEventLogs and they are no more processed.
They don’t exist in metrics table in DB.
This SQL command gives the following result:
SELECT * FROM scheduled_tasks where class_name=‘plugins.generic.usageStats.UsageStatsLoader’
plugins.generic.usageStats.UsageStatsLoader 2019-04-30 07:33:19
So UsageStatsLoader seems to be executed
We don’t change our config file:
scheduled_tasks = On
Acron plugin is activated
It’s the first time it’s happen even when we upgraded from OJS2 to OJS3
I installed the OJS 3.1.2 released by downloading the whole archive from the official website.
The previous release installed was OJS 188.8.131.52
usageStats is not present in “lib/pkp/plugins/generic/” directory and is only present in “plugins/generic/” directory.
I searched for other occurence of geoip_country_code_by_name_v6 in OJS home directory but I didn’t find any.
I never tested to execute manually this command in previous release:
php tools/runScheduledTasks.php plugins/generic/usageStats/scheduledTasks.xml
Is it the good way to do execute it ?
What will it do exactly?
Does this relate to the fact that usage_events_*.log files stay in directory usageStats/usageEventLogs and they are no more processed ?
OJS stores log files initially in the “usageEventLogs” directory. The log files must be moved to a “stage” directory before being processed (in the “processing” directory) into either the “archive” directory (for success), or the “reject” directory (for failure).
Running the command php tools/runScheduledTasks.php plugins/generic/usageStats/scheduledTasks.xml will process any OJS log files which have been moved to the “stage” directory.
You probably instead want to run the command for autostaging: php tools/runScheduledTasks.php plugins/generic/usageStats/scheduledTasksAutoStage.xml
This will move the logs into the “stage” directory automatically for processing.
The last log file in “/var/files/usageStats/archive” directory is from the day before the upgrade to OJS 3.1.2
The log files in “/var/files/usageStats/usageEventLogs” directory stayed in this directory since the day of the upgrade to OJS 3.1.2
There are not log files in “/var/files/usageStats/processing”, “/var/files/usageStats/stage”, “/var/files/usageStats/reject” directories when I looked
When I run the command for autostaging:
php tools/runScheduledTasks.php plugins/generic/usageStats/scheduledTasksAutoStage.xml
I noticed when I removed the entry in the DB table scheduled_tasks with the class_name = plugins.generic.usageStats.UsageStatsLoader
And then I ran again the command for autostaging, I have the same error when I run the usageStats/scheduledTasks.xml command:
PHP Fatal error: Cannot redeclare geoip_country_code_by_name_v6() in /ojs_home/plugins/generic/usageStats/lib/geoIp/geoip.inc on line 430
I don’t know why I have got this error and if it’s the reason why logs are not moved into the “stage” directory automatically for processing.
I also noticed the time registered in scheduled_tasks for the execution of plugins.generic.usageStats.UsageStatsLoader is not the same that the time server (2 hours difference). I don’t know if it’s relevant. It was maybe the case before.
I don’t know what is wrong, where I have to looking for, and what can I do as tests.
The scheduledTasksLogs directory should exist directly under your files_dir, parallel to the usageStats directory, which contains archive, processing, reject, etc.
The lines from the metrics table reference metrics ingested from event logs from 2016-05-12. Do you see recent entries as well?
You generally want to choose to either run the “tools/runScheduledTasks.php” command in cron, or use the acron plugin, which will automatically run the “scheduledTasksAutoStage” task. Do you have the acron plugin enabled?
The usage statistics processing is rate-limited and will not immediately run again unless the time indicated in the scheduled_tasks table has expired. If you clear the entry from the scheduled_tasks table and the command runs again, this is due to the rate limiting.
This is probably related to the value of the PHP date.timezone configuration. It should not cause a failure of processing the usage statistics, but you should set the value in your php.ini (or set time_zone in config.inc.php).
The PHP Fatal error is a concern. Since you have searched for geoip_country_code_by_name_v6 within the OJS codebase and only found the one occurrence, one possibility is that the geoip.inc file is being included twice somehow. If this were the case, however, it should error on the first function in that file (geoip_load_shared_mem), not geoip_country_code_by_name_v6.
The scheduledTasksLogs directory do exists under my files_dir, parallel to the usageStats directory.
The last Usagestatisticsfileloadertask*.log is the day of upgrading to OJS 3.1.2. It’s written inside that the “/var/files/usageStats/processing/usage_events_yyyymmdd.log” was processed and archived.
yyyymmdd is the date of the day before upgrading to OJS 3.1.2
The last records in metrics table are with the last usage_events_yyyymmdd.log file stored in archive directory with the date of the day before upgrading to OJS 3.1.2
Processing usage events logs stopped when I upgraded to OJS 3.1.2 but I don’t know why.
and it was immediately filled with serveral log files. The log file for the usage statistics contains several warnings, that the usage_events_log file contains an adress where the system can’t remove the base url. What does that mean?
[2019-05-09 11:25:21] [Warnung] Die Zeile 7914 aus der Datei /opt/ojs-mthz-files/usageStats/processing/usage_events_20190508.log enthält eine Adresse, aus der das System nicht den Basis-URL entfernen kann.
The metrics table doesn’t contain recents entries, although there are daily event log-files in the archive-folder since 2016 up to today.
The acron plugin is enabled. I just restarted it and the event log files was re-generated, but the reports and counter statistics are still empty:
Do you have an more idea what goes wrong? We need theses statistics on monday…
Does line 7914 of usage_events_20190508.log look like an OJS URL? If so, it probably means that OJS failed to process the line. This can happen if OJS can’t correctly determine the hostname and any path for the URL. For example, if the line reads:
But if your config.inc.php references a base_url for something other than http://dev.ojs.hostname.tld/ojs3, then OJS may be failing to process that particular line. If all lines reference this warning, none of the lines are actually processed.
If you can correct the base_url settings in config.inc.php you can re-process the archived log files by moving them back from the “archive” folder to the “stage” folder. Try it with one file first and (if successfully processed) move all of the files from the missing date range. Reprocessing the files will overwrite any existing rows in the metrics table which match that load_id.
I still have the same problems. I tried to looking for what happened when I execute the scheduledTasks commands by displaying some messages in the code step by step:
When I execute the command: php tools/runScheduledTasks.php plugins/generic/usageStats/scheduledTasks.xml
I have this error: PHP Fatal error: Cannot redeclare geoip_country_code_by_name_v6() in /ojs_home/plugins/generic/usageStats/lib/geoIp/geoip.inc on line 430
I found the error occurs when in file : /ojs_home/plugins/generic/usageStats/UsageStatsLoader.inc.php
this command line is executed: $geoLocationTool = $statsHelper->getGeoLocationTool();
I found that function “getGeoLocationTool()” is called in this file: /ojs_home/lib/pkp/classes/statistics/PKPStatisticsHelper.inc.php
And the error occurs after this command line is executed: $geoLocationTool = $plugin->getGeoLocationTool();
When I execute the command: php tools/runScheduledTasks.php plugins/generic/usageStats/scheduledTasksAutoStage.xml
I found that in the following file: /ojs_home/lib/pkp/classes/cliTool/ScheduledTaskTool.inc.php
In function “parseTasks”
When “$frequency” variable is tested, it has got a value, and this value is tested with this function: $canExecute = ScheduledTaskHelper::checkFrequency($className, $frequency);
The checkFrequency “function” is included in this file: /ojs_home/lib/pkp/classes/scheduledTask/ScheduledTaskHelper.inc.php
“$frequency” variable is an array which not contains month or day. But it contains an hour value of “0” which is consider not valid: [attributes] => Array([hour] => 0)
That’s the reason why, “$canExecute” variable is false and nothing is execute after
Do you have any idea why in the 1st command, the call of “plugin->getGeoLocationTool()” command generates the following error ?: PHP Fatal error: Cannot redeclare geoip_country_code_by_name_v6() in /ojs_home/plugins/generic/usageStats/lib/geoIp/geoip.inc on line 430
And why in the 2nd command, the presence of the “hour” parameter with the value “0” in “$frequency” variable generates an error ?
Where does the “hour” parameter come from? And where can I change it ?
This checks to see if the current hour has passed the hour of the last runtime. Thus, the scheduledTasksAutoStage.xml can only be run once a day.
I’m not able to reproduce the problem of Cannot redeclare geoip_country_code_by_name_v6() in my install of 3.1.2 stable. I’m very curious about what is causing this. If you search for ‘geoip.inc’ in your OJS files, do you find any references other than:
$ grep -r 'geoip.inc' *
plugins/generic/usageStats/lib/geoIp/README:Just place the 'geoip.inc' file somewhere according to the 'include_path' directive of
plugins/generic/usageStats/lib/geoIp/geoipcity.inc: * Changed inclusion of geoip.inc file to require_once, so that
plugins/generic/usageStats/lib/geoIp/geoipcity.inc: * this library can be used in the same script as geoip.inc.