[OJS 3.1] How Usage Statistics Plugin get the information to show in a graph?

juancure · March 5, 2018, 6:54pm

Hi everyone

I have enable the Usage Statistics plugin for my OJS 3.1 journal, and I have noticed that for the last published issue’s articles in the details article page, a bar graph has appeared with the downloads information. However for the articles in previous issues, that graph is not available.

I have the doubt how the usage statistics plugin look for the downloads information? and I have noticed that in the database there is a table called usage_stats_temporary_records but this tables is empty.

I would like to add about the number of visits to our journal. This plugin could provide this information?

Well I hope someone could give some information about these doubts.

bozana · March 6, 2018, 3:47pm

Hi @juancure

When enabled, the Usage Statistics plugin will track the access to the article page, galley page/download, issue page and journal home page. This plugin considers the standard COUNTER. Thus the other visits, to other journal pages are not considered here.
The access is then logged in the files. Those log files are in your files folder, in the usageStats sub-folder. Those log files are then processed and the view counts calculated and saved in the DB table metrics. (First, the new log files are in the folder usageEventLogs. When being processed they are moved to stage, then to processing, and then either to reject or archive folder – depending on if any error occurred i.e. if the file was successfully processed.) .
For the graph that is displayed on the article view page, the information in the DB table metrics is used. Only the downloads of the article galley files are considered for the graph. In the current OJS version, this graph only considers the view counts from the current year. This will change with the next OJS release, so that the graph will consider the view counts from the last 12 months, s. Display usage statistics for last 12 months instead of for current year · Issue #3283 · pkp/pkp-lib · GitHub.

Thus, you could double check if everything is OK with the data in the DB table metrics, or with your log files. Maybe those other article files were not downloaded in this year?

See also this document for more information: https://pkp.sfu.ca/wiki/index.php?title=PKP_Statistics_Framework

Best,
Bozana

juancure · March 7, 2018, 6:33pm

Hi again @bozana

Thanks a lot for your response, Again you help me to resolve my doubts. However I have a doubt yet. The Usage Statistics Plugin doesn’t generate the calculated number of download on the fly?. I mean if in this moment I would download 3 times the PDF of an article, this new number of downloads doesn’t appear inmediatly in the graph? If it is’nt that way. How long does it take to render that new data in the graph?

I have thought that the plugin doesn’t show the graph for my articles of previous issue, because that issue were published over an OJS 3.0.1, while the articles which have that graph were published in an upgraded OJS 3.0.2?

bozana · March 8, 2018, 1:10pm

Hi @juancure

The usage statistics log files are processed every day and the processing is triggered by running the appropriate scheduled task. The scheduled task is run either by defining a cron job on your server or if you use the Acron plugin automatically when a journal page is accessed. I suppose you are using the Acron plugin – is it enabled?
And yes, that means that the counts are visible in the graph with ca. one day delay.

It would be best if you could double check your usage statistics log files – if they are processed successfully and in the folder archive. Maybe also if they contain the URLs of galleys of those problematic articles with no graph. And to double check your DB table metrics – if there are any entries for those galley of the articled that have no graph displayed.

The change to 3.0.2 should not matter – the usage has been logged (and processed) for all articles from the time when you correctly set up/enabled the usage statistics plugin.

Best,
Bozana

juancure · February 21, 2019, 12:22am

Hi @bozana

I’m sorry because I did not follow with this topic all this time and again I come with this for resolve my case particular with the usage statistics plugin. The issue is that my first published issue which was in OJS 3.0.1.0, the Usage Statistics plugin does not display the download data chart and only there is a message as Download data is not yet available instead:

%5B1%5DDownload_Data_Is_Not_Yet_Available

In your last reply you mentioned that the plugin only display a chart with download data for the current year, I have noticed that for all the other articles that chart is displayed, the fail is only with the articles in the first published issue.

I have reviewed some files usage_events_2018XXXX.log within the subdirectory archive, only this and usageEventLogs directory have files and I have noticed there are URL’s for those articles which do not have the download chart.

The problem with this URL’s is that some of them are no the final URL, this happen because in the beginning of our Journal our production server did not have a server name established only IP, after we got the server name and the URL’s in that files, starting to have the server name and in other cases we called the OJS installation directory as ojs3, ojs3_1 or merely ojs resulting some URL’s for the same article as:

In this moment the correct URL for that article is the number 1. however the number 3 is correct too. Well it’s true that all the new URL’s added to the log file within usageStats/usageEventLogs have the server name. My doubt is: all that URL’s that do not match with the final URL for the article could be considered for the aggregated download metric if I try to reprocessing all the log files within archive directory again?

I hope you can help me to find a solution to this trouble. Thanks in advance.

bozana · February 21, 2019, 9:41am

Hi @juancure

Yes, the usage statistics plugin looks in the config.inc.php and uses that base_url to match the URLs in the log file when processing the log file. Thus, in order to be considered your URLs in the log files should be the same as in config.inc.php. If this is not the case, maybe you would like to repair/change them in the log files.
The other thing you could take a look, if you see those articles, from those older log files from the beginning in your DB table metrics. Eventually your base_url in the config.inc.php at that time was the same and the files were processed back then. It does not look like that, but… It is good to double check the metrics table as well and what files and articles are there – this is the result that the graph is then using.
If you would like/need to reprocess the log files, you would need to adapt the URL to match the one in config.inc.php and move them in the folder usageEventLogs/. From there they will be considered for processing next time your scheduled task is executed.

Best,
Bozana

juancure · February 22, 2019, 12:19am

Hi @bozana thanks a lot for your response, despite of this is an old topic.

Well I have read the Appendix C: Processing Log Files so I disabled the Acron plugin and I try to reprocess the log files using a cron job as the following

* * 0 0 0 /usr/bin/php tools/runScheduledTasks.php lib/pkp/plugins/generic/usageStats/scheduledTasks.xml

But the scheduledTask never executed. That’s why I have enabled the Acron plugin again and I have run the Reload Scheduled Tasks option. This action reprocess the log files moved to the folder usageEventLogs as you said and not to the folder stage as the documentation said. However this only has effect in the same articles which the download data chart was displayed before, but in those articles which it was not displayed remain the same.

Moreover when I generate a report using the View Report, the resulting CSV file has the Total Galley Views as zero for all these articles which the chart is not displayed.

I have reviewed the metrics table in my DB with the following query:

SELECT load_id, assoc_type, assoc_id, submission_id, day, month, file_type, metric_type, metric, pkp_section_id, assoc_object_id from metrics where submission_id=8;

with the submission_id as the id for all these articles and I can look resulting rows as the following:

 load_id          | assoc_type | assoc_id | submission_id |   day    | month  | file_type | metric_type  | metric | pkp_section_id | assoc_object_id 
---------------------------+------------+----------+---------------+----------+--------+-----------+--------------+--------+----------------+-----------------
 usage_events_20170119.log |        515 |       40 |             8 | 20170119 | 201701 |         2 | ojs::counter |      1 |              1 |               1
 usage_events_20170119.log |        515 |       41 |             8 | 20170119 | 201701 |         3 | ojs::counter |      3 |              1 |               1
 usage_events_20180123.log |    1048585 |        8 |             8 | 20180123 | 201801 |           | ojs::counter |      6 |              1 |               1
 usage_events_20170120.log |        515 |       40 |             8 | 20170120 | 201701 |         2 | ojs::counter |      1 |              1 |               1
 usage_events_20180123.log |        531 |       45 |             8 | 20180123 | 201801 |         3 | ojs::counter |      3 |              1 |               1
 usage_events_20180123.log |        531 |      115 |             8 | 20180123 | 201801 |         2 | ojs::counter |      2 |              1 |               1
 usage_events_20170124.log |        515 |       40 |             8 | 20170124 | 201701 |         2 | ojs::counter |      1 |              1 |               1
 usage_events_20180227.log |        531 |      109 |             8 | 20180227 | 201802 |         2 | ojs::counter |      1 |              1 |               1
 usage_events_20180124.log |    1048585 |        8 |             8 | 20180124 | 201801 |           | ojs::counter |      1 |              1 |               1
 usage_events_20170127.log |        515 |       40 |             8 | 20170127 | 201701 |         2 | ojs::counter |      1 |              1 |               1
 usage_events_20180227.log |    1048585 |        8 |             8 | 20180227 | 201802 |           | ojs::counter |      1 |              1 |               1
 usage_events_20170206.log |        515 |       40 |             8 | 20170206 | 201702 |         2 | ojs::counter |      1 |              1 |               1
 usage_events_20170207.log |        515 |       40 |             8 | 20170207 | 201702 |         2 | ojs::counter |      1 |              1 |               1
 usage_events_20180125.log |        531 |      115 |             8 | 20180125 | 201801 |         2 | ojs::counter |      1 |              1 |               1
 usage_events_20170208.log |        515 |       40 |             8 | 20170208 | 201702 |         2 | ojs::counter |      2 |              1 |               1
 usage_events_20180126.log |    1048585 |        8 |             8 | 20180126 | 201801 |           | ojs::counter |      1 |              1 |               1
 usage_events_20170222.log |        515 |       40 |             8 | 20170222 | 201702 |         2 | ojs::counter |      2 |              1 |               1
 usage_events_20180228.log |        531 |       45 |             8 | 20180228 | 201802 |         3 | ojs::counter |      1 |              1 |               1
 usage_events_20180228.log |    1048585 |        8 |             8 | 20180228 | 201802 |           | ojs::counter |      3 |              1 |               1
 usage_events_20180228.log |        531 |      109 |             8 | 20180228 | 201802 |         2 | ojs::counter |      2 |              1 |               1
 usage_events_20180129.log |        531 |      115 |             8 | 20180129 | 201801 |         2 | ojs::counter |      1 |              1 |               1
 usage_events_20180129.log |    1048585 |        8 |             8 | 20180129 | 201801 |           | ojs::counter |      1 |              1 |               1
 usage_events_20180301.log |    1048585 |        8 |             8 | 20180301 | 201803 |           | ojs::counter |      4 |              1 |               1
 usage_events_20180301.log |        531 |      115 |             8 | 20180301 | 201803 |         2 | ojs::counter |      1 |              1 |               1
 usage_events_20170330.log |        515 |       40 |             8 | 20170330 | 201703 |         2 | ojs::counter |      5 |              1 |               1
 usage_events_20170331.log |        515 |       40 |             8 | 20170331 | 201703 |         2 | ojs::counter |      3 |              1 |               1
 usage_events_20180130.log |        531 |      109 |             8 | 20180130 | 201801 |         2 | ojs::counter |      1 |              1 |               1
 usage_events_20170402.log |        515 |       40 |             8 | 20170402 | 201704 |         2 | ojs::counter |      1 |              1 |               1
 usage_events_20180130.log |    1048585 |        8 |             8 | 20180130 | 201801 |           | ojs::counter |      2 |              1 |               1
 usage_events_20180301.log |        531 |      109 |             8 | 20180301 | 201803 |         2 | ojs::counter |      3 |              1 |               1
 usage_events_20180131.log |    1048585 |        8 |             8 | 20180131 | 201801 |           | ojs::counter |      1 |              1 |               1
 usage_events_20180211.log |    1048585 |        8 |             8 | 20180211 | 201802 |           | ojs::counter |      1 |              1 |               1
 usage_events_20180301.log |        531 |       45 |             8 | 20180301 | 201803 |         3 | ojs::counter |      1 |              1 |               1
 usage_events_20180201.log |        531 |      109 |             8 | 20180201 | 201802 |         2 | ojs::counter |      2 |              1 |               1
 usage_events_20180201.log |    1048585 |        8 |             8 | 20180201 | 201802 |           | ojs::counter |      5 |              1 |               1
 usage_events_20180202.log |        531 |      115 |             8 | 20180202 | 201802 |         2 | ojs::counter |      7 |              1 |               1
 usage_events_20180202.log |    1048585 |        8 |             8 | 20180202 | 201802 |           | ojs::counter |      6 |              1 |               1
 usage_events_20180302.log |    1048585 |        8 |             8 | 20180302 | 201803 |           | ojs::counter |      1 |              1 |               1
 usage_events_20180302.log |        531 |       45 |             8 | 20180302 | 201803 |         3 | ojs::counter |      1 |              1 |               1
 usage_events_20180302.log |        531 |      115 |             8 | 20180302 | 201803 |         2 | ojs::counter |      1 |              1 |               1
 usage_events_20180309.log |        531 |       45 |             8 | 20180309 | 201803 |         3 | ojs::counter |      7 |              1 |               1
 usage_events_20180309.log |    1048585 |        8 |             8 | 20180309 | 201803 |           | ojs::counter |     15 |              1 |               1
 usage_events_20180309.log |        531 |      109 |             8 | 20180309 | 201803 |         2 | ojs::counter |      7 |              1 |               1
 usage_events_20180205.log |        531 |      115 |             8 | 20180205 | 201802 |         2 | ojs::counter |      2 |              1 |               1
 usage_events_20180205.log |    1048585 |        8 |             8 | 20180205 | 201802 |           | ojs::counter |      4 |              1 |               1
 usage_events_20180303.log |        531 |      109 |             8 | 20180303 | 201803 |         2 | ojs::counter |      2 |              1 |               1
:

That’s why I cant understand the reason why the download data chart is not displayed.

Now It’s important to mention that when I created the file galleys for my first issue, I have created them as supplementary files to the articles. It’s possible that the Usage Statistics Plugin does not count the metrics for this kind of file galleys? It’s the only galleys created as supplementary and which have the problem.

juancure · February 22, 2019, 1:24am

Hi @bozana

i have realized through a test that my articles for which the download data chart is not displayed is because all their galleys of file kind were created as supplementary galleys. I have created a file galley of the article text kind for one of my articles with the problem, then I have looked the new galley and in this way the system will count it as one download.

Then I have moved the usage_event_log file for the current day to the folder stage and now I have run correctly the command for executing scheduledTasks the following way:

php tools/runScheduledTasks.php lib/pkp/plugins/generic/usageStats/scheduledTasks.xml

This generated the information for DB metrics table and finally I have looked the download data chart for my article

My doubt now is how can I turn my PDF, HTML, XML galleys created as supplementary galleys to article text galleys?

bozana · February 27, 2019, 12:07pm

Hi @juancure

Oh, yes, only the full text files/galleys are considered in the usage statistics plugin.
Unfortunately, the change of supp files to galleys is not so easy:
– if a file is a supp file or galley depends on the used genre. You can see the existing genres in your DB table genres. The galleys are not supplementary and not dependent, e.g. “Article Text” i.e. the genre with the entry_key = SUBMISSION in the DB table genres.
– if a file is a supp file, there will also be an entry in the DB table submission_supplementary_files for it.
– if a file is a galley, there will also be an entry in the DB table submission_galleys and eventually submission_galley_settings for it.
– there are also submission_artwork_files.
– galleys will have the appropriate genre_id in the DB table submission_files, as well as assoc_type = 521 and the appropriate assoc_id = galley_id (from the DB table submission_galleys).
– also the submission_file_settings will be different for galleys and supp files.

Thus, pro file you would need to add an entry in the table submission_galleys and eventually in the submission_galley_settings, to adapt the DB table submission_files to enter the correct genre_id, assoc_type and assoc_id, as well as submission_file_settings table. Then you would also need to remove the entry from the table submission_supplementary_files.

How many files are ‘wrong’?

Best,
Bozana

juancure · February 28, 2019, 6:11am

Hi @bozana thanks a lot for your response.

Well I have about 15 galleys, that when they were created I chose the option “Other” in the article component catalog, i.e the genre with entry_key = OTHER.

My genres table in my DB is shown in the following image:
It seems there are more genres created for the user.
I have the way to recognize all the rows in my table submission_galleys for which the genre with the entry_key = OTHER
In my table submission_supplementary_files I have noticed that there are about 216 rows.
In my table submission_galley_settings there are about 130 rows, I left an image with a chunk of the result
My table submission_artwork_files is empty

I believe that I have the way to make the change between all this files supplementaries to galleys in the DATABASE level, but I have the doubt if I have to move and rename alll these files within the files_dir directory?

bozana · February 28, 2019, 10:08am

Hi @juancure,

OK, from what I see, I think you will not have to do any changes in submission_galleys and submission_galley_settings tables. You have the entries for those 15 files in the table submission_galleys, correct?
In that case the table submission_files will also contain the correct values for assoc_type (521) and assoc_id (the appropriate galley_id) for those 15 files, correct?
If so, then you would only need to:
– change the genre_id (to e.g. 1) in the table submission_files.
(I cannot see what is the genre_id for the genre OTHER in your table genres. And I do not know/understand what are those other genres (with no entry_key) – a custom ones? Thus, please check what is the correct genre_id that you should use – e.g. what genre_id have the other, correct galleys)
– rename the file in the files folder. You will find the files in the subfolder journals//articles//submission/proof/. The file name is generated like this: pkp-lib/SubmissionFile.inc.php at omp-3_1_1-4 · pkp/pkp-lib · GitHub. Thus, it is submissionId-genreId-fileId… You would only need to change the genreId part (to e.g. 1).
– check now that everything is OK, that you can download the galley via UI as editor and that you can see it as reader on the article view page.
– remove all setting_names except the setting_name = ‘name’ from the table submission_file_settings for those 15 files.
– remove the entry from the submission_supplementary_files.

Do this file-by-file and be sure/double check that everything is correct with one file before you proceed with the other file.

Best,
Bozana

juancure · March 5, 2019, 12:57am

Hi @bozana

I write you again only to tell you that I have accomplished it, I have an Usage Statistics chart with downloads data for my articles which the chart was not displayed.

I have followed your tips in your previous response. I left as sample the process to convert a supplementary galley from a submssion to primary galley, this is the scenario:

I have a submission of an article, the submission_id=8. For this submission I have a galley which was created with the “Other” article component option. This galley has the galley_id=30 and genre_id=12. The file associated to this galley was an XML which has the file_id=45. well the goal is turn this supplementary galley to submission (primary) galley. The working process is composed with the following steps:

Database Level

In the submission_files table:

UPDATE submission_files SET genre_id=1 WHERE submission_id=8 and file_id=45;

In the submission_file_settings table:

DELETE FROM submission_file_settings WHERE file_id=45 and setting_name!='name';

In the submission_supplementary_files table:

DELETE FROM submission_supplementary_files WHERE file_id=45;

Filesystem level

In the files_dir directory, I have renamed all the files associated with the galley. I have noticed that each time you use the change file from Administrator interface, this action uploads the file keep the previous file up. Well because we used multiple times that action, I have 31 different version for this file in my files_dir/journals/1/articles/8/submission/proof. Well I could rename file per file, but I have found a easy way using the following command:

for f in *.xml; do mv "$f" "$(echo "$f" | sed s/12/1/)"; done

The rename of files only consists in change from 8-12-45-11-10-20170406.xml to 8-1-45-11-10-20170406.xml the genre_id from 12 to 1.

Well after run the command to reload the scheduled task associated with the statistics:

php tools/runScheduledTasks.php lib/pkp/plugins/generic/usageStats/scheduledTasks.xml

And clean all the template cache generated I have got the following chart:

%5B6%5DDescargas_chart_with_data_again

Thansk a lot for your help @bozana

Regards best