I am writing on behalf of the Milano University Press to present and ask for comments about a caching mechanism for OJS and OMP that we are launching with our technology partner Archimede Informatica (forum registered users: @razzi, @paolo_pellegrino, @monmarzia), and which we hope will be of interest to all users and therefore become part of future releases of the applications.
The idea of working on caching stems from the fact that our multijournal OJS has recently faced some difficulties related to its size. We experienced a decline in performance due to known issues such as inefficiencies in the management of the database underlying the applications by the core code or some plugins, which we have therefore had to abandon. This is exacerbated by the growing traffic on websites generated by massive indexing by search engine bots, including for the purpose of training artificial intelligence.
This was also discussed in the recent call on OJS development on December 15, and there is no shortage of forum posts on the subject, such as this one.
We are aware that
The problem to be solved:
Multi-journal OJS instances with a high number of journals and articles (>30 journals) have long response times when serving a page, both in the front-end and backend
Response times increase significantly when using certain features: OJS search, Issue TOC page, and some external plugins (e.g., “Recommend Articles by Author”) or system plugins (“Usage Statistics”, which is essential to satisfy DOAS recommendation 6.3: https://zenodo.org/records/15147823 )
The slowness is caused by SQL queries executed on the database (querys on very huge tables and/or thousands of queries to serve a single page), and cannot be definitively resolved simply by improving DBMS performance through database query optimization. (Database query optimization does not conflict with our caching system; rather, it can contribute to further improvements in front-end response times, but above all to reducing back-end response times and server load.)
Objectives:
Drastically reduce front-end response times in multi-journal OJS instances with a high number of journals and articles (>30 journals).
The solution must work starting from OJS 3.5, while keeping 3.6 in mind, at least by beginning to understand how much and in what way it can be adapted for OJS 3.6
The solution should ideally be reusable in later OJS versions.
How to achieve this:
Implement a caching mechanism for the OJS front-end that dratsically reduce the number of SQL queries executed every time OJS serves a front-end page.
Caching should operate at the HTML page level (each HTML page served by OJS in the front-end).
Caching should be implemented at the server level, not at the browser level (although browser caching optimizations do not conflict with our caching system; rather, they can help further improve response times and reduce server load).
Possible approaches:
based on static HTML files generated by OJS on the server, similar to the feature already present in OJS 3.3, which can be enabled in the config.inc.php file by setting `web_cache = On`. The current feature has a serious drawback: OJS does not automatically update the static HTML files when the content in OJS changes. You can only configure the cache duration, and the cache is completely reset when it expires.
based on the use of the database, by using caching tables to store the static HTML of the frontend pages. These pages will-be served by OJS instead of re-creating it every time by querying the DB.
Regardless of the chosen approach (static files vs database), the solution must include a backend mechanism that automatically removes cached pages whenever content related to one or more front-end pages is updated. These pages should be re-cached as soon as a new front-end request for them arrives.
The caching mechanism must be compatible with OJS APIs and other APIs enabled through OJS plugins (e.g., OAI-PMH, PKP-PN, etc.).
The caching mechanism must be compatible with the OJS access log system used by the Usage Statistics plugin to generate statistical reports.
We believe solutions based on caching systems external to OJS (Varnish, Redis, nginx) are not well-suited for the following reasons:
high configuration and maintenance complexity (Installing and configuring Varnish, Redis, nginx)
difficulty managing cache invalidation toward external caching tools for pages whose content is modified in the OJS backend
incompatibility with OJS access log recording used by the Usage Statistics plugin
The solution must take into account the fact that query optimisation can solve some problems of performance; we are available to work in that direction too; however, we believe that an efficient caching system can play a key role for large-scale installations with a large number of journals, together with database optimisation like this
the solution should not add a layer of complexity and have to be usable for all the community, being at least a feature increasing frontend performance; this is one of the reason we think to start from TYPO3 CMS caching mechanism, in other words we want OJS/OMP frontend having the caching features of a robust website
We’re aware of a few specific performance bottlenecks – the Recommend by Author plugin, for example, has an issue filed for performance improvements. I suspect a file-based cache for that plugin would be a relatively small change.
Note that we’ve rewritten the search engine implementation for release in OJS/OMP/OPS 3.6 – see details here. This adds support for OpenSearch but also rewrites the database-based search engine (which is what pretty much everyone will be using). It should perform a lot better.
Beyond that, I see some specific opportunities:
There are probably some basic improvements to be made in reducing the number of database queries required to build a page. We’ll be doing some of this work ourselves during the 3.6 dev cycle (over the summer).
There are also some basic improvements to be made in the headers OJS/OMP/OPS generate, in order to instruct both remote browsers and caching proxies about the cacheability of pages.
It’s also possible to reintroduce object and static page caching into the codebase (the old web_cache and object_cache settings in config.inc.php) – but I’m less enthusiastic about putting time into these, as they can quickly grow complicated and I would rather see the effort go into improving baseline performance first.
Because a lot has already changed since 3.3 and 3.4, I’m hoping you’re interested in working with at least OJS/OMP/OPS 3.5; I think you’ll find it’s already a much more modern coding experience, and it would be hard to forward-port improvements made to earlier branches!
How much of the trouble you experienced with performance was related to specific bottlenecks like the “recommend by author” plugin?
Regards,
Alec Smecher
Public Knowledge Project Team
Hi Alec.
Here are the answers to some of your questions.
OJS 3.3 and 3.4 will reach end-of-life (EOL) in about a year. Milan University Press is currently using OJS 3.3 but has already planned to upgrade to OJS 3.5 as soon as the LTS version is released. On the other hand, Milan University Press will not adopt OJS 3.6 since it will not be an LTS release and therefore will not be able to benefit from the performance improvements that will be introduced in that version.
This is why Milan University Press intends to focus its work on this new feature for OJS 3.5. From what we have understood so far, it would not be too difficult to carry over the work done for 3.5 to 3.6.
Regarding performance issues, the most serious ones are certainly linked to certain plugins, but there are also problems with core features, such as the issue index and article search, for which improvements are planned in 3.6.
We agree that performance improvements must also be made at the query optimization level and that these can improve the current situation. Moreover, these are inevitable for improving performance in the OJS backend. However, we believe that for the frontend, a caching mechanism like the one we have in mind would introduce drastic improvements for large OJS7OMP installations.
Regarding performance issues, I can share the plugins below (there are releases for OJS/OMP/OPS 3.3/3.4/3.5), which we’re using at the PKP Publishing Services:
Front End Cache: This one caches the front end pages at the server, and also at the client level, through cache headers, without disrupting the statistics (a hit at the server is generated when needed). It’s a site-wide plugin, so once enabled, the whole installation should benefit from it.
At the 3.3 release, it also has a kind of experimental code (we’re using it and so far nobody has complained) to improve the database requests (i.e. instead of hundreds of queries, a single one, to eager load data, is issued).
Full Text Search: This plugin is basically a simplified implementation of the Improve search framework · Issue #8920 · pkp/pkp-lib · GitHub which is also a site-wide plugin, it has a limited life span (at 3.6 the search performance is a solved problem, so I’m not going to port it).
I can confirm both together highly improve the performance of the application.
Our application has many reads and few writes, therefore our focus must be on the cache. With a great cache system in place, we can delay generic performance enhancements and concentrate on the bad routes (anything that takes seconds to run).
The plugin I shared caches the whole page, but that’s not the right approach… The front end must be able to cache small pieces of the application (e.g. header, list of articles, menu, …) independently and in a reactive way. By reactive, I mean the cache of the menu should be invalidated only if the menu data gets updated.
Thank you @jonasraoni for your feedback and for sharing the plugins.
We’ll test them on the OJS and OMP platforms at Milano University Press and let you know how the tests went.
Thank you also for your thoughts on the frontend cache. If I understand your position correctly, you agree that investing time and effort in a robust caching system is worthwhile.