OJS 2.4.3: Catastrophic slow query in OAIDAO

Hi, running over 500 journals on a single installation, so at all times there is pressure on mysql, and any bottleneck in database connections tends to be bad news.

The example query below from /classes/oai/ojs/OAIDAO.inc.php seems to appear in various permutations in the database connection queue whenever connections time out … this query can take up to 120 seconds to execute, which is pretty disastrous! I’ve tried various indexes in the database to alleviate the load, but to no avail. Any suggestions?

SELECT CASE WHEN COALESCE(dot.date_deleted, a.last_modified) < i.last_modified THEN i.last_modified ELSE COALESCE(dot.date_deleted, a.last_modified) END AS last_modified,
COALESCE(a.article_id, dot.data_object_id) AS article_id,
COALESCE(j.journal_id, tsoj.assoc_id) AS journal_id,
COALESCE(tsos.assoc_id, s.section_id) AS section_id,
i.issue_id,
dot.tombstone_id,
dot.set_spec,
dot.oai_identifier FROM mutex m LEFT JOIN published_articles pa ON (m.i=0)
LEFT JOIN articles a ON (a.article_id = pa.article_id)
LEFT JOIN issues i ON (i.issue_id = pa.issue_id)
LEFT JOIN sections s ON (s.section_id = a.section_id)
LEFT JOIN journals j ON (j.journal_id = a.journal_id)
LEFT JOIN data_object_tombstones dot ON (m.i = 1)
LEFT JOIN data_object_tombstone_oai_set_objects tsoj ON tsoj.assoc_id = null LEFT JOIN data_object_tombstone_oai_set_objects tsos ON tsos.assoc_id = null WHERE ((s.section_id IS NOT NULL AND i.published = 1 AND j.enabled = 1 AND a.status <> 0) OR dot.data_object_id IS NOT NULL) AND CASE WHEN COALESCE(dot.date_deleted, a.last_modified) < i.last_modified THEN (i.last_modified >= ‘2016-06-05 02:00:00’) ELSE ((dot.date_deleted IS NOT NULL AND dot.date_deleted >= ‘2016-06-05 02:00:00’) OR (dot.date_deleted IS NULL AND a.last_modified >= ‘2016-06-05 02:00:00’)) END ORDER BY journal_id;

Hi @makouvlei,

Is it an option to upgrade to the latest OJS 2.x release? OJS 2.4.3 is quite old. I can’t promise that it’ll resolve the issue but it’s possible.

Have you worked with the SQL EXPLAIN tool to determine why this query takes so long?

Regards,
Alec Smecher
Public Knowledge Project Team

Hi Alec, I can at some point upgrade to latest 2.x, although probably not for some time, as that’s not a trivial matter, having an old and very large OJS installation, which also includes quite a bit of customisation. We have applied the recommended patches though.

For now here’s 3 screenshots

  • the output of the EXPLAIN query

  • and further below some stats generated by Navicat when executing the query on my testing server (where it takes 270 sec)

Any help will be appreciated.

Regards
Jannie van Tonder
AJOL

Hi @makouvlei,

At a glance, those cardinalities don’t look amiss to me… Do you have someone on hand who can try removing joins from the query until it performs as expected? One necessary weirdness in that query is the mutex table – its use is explained here as a way to avoid UNION joins, which aren’t supported in all versions of MySQL that OJS 2.4.x supports.

Regards,
Alec Smecher
Public Knowledge Project Team

Hi Alec,

I’ll have a go at removing joins and see what I come up with.

We’re using MySQL 5.7.17. How would you code it without the mutex?

Regards
Jannie

Removing data_object_tombstones table, execution time drops below 1 second.

Hi @makouvlei,

I’ve just rewritten those queries in the OJS 3.x / OMP 3.x codebase to use UNION instead of the mutex table work-around – see pkp/pkp-lib#2407 Remove mutex work-around for UNION queries by asmecher · Pull Request #2408 · pkp/pkp-lib · GitHub (lib/pkp submodule) and pkp/pkp-lib#2407 Remove mutex work-around for UNION queries by asmecher · Pull Request #1344 · pkp/ojs · GitHub (OJS repo). These will need back-porting to OJS 2.x; we’re unlikely to do that because it would require a significant bump to the supported DBMS versions, but you’re welcome to give it a crack. (Note that those are not merged yet – they still need code review – but test OK locally for me.)

Regards,
Alec Smecher
Public Knowledge Project Team

Thanks Alec, much appreciated, I’ll certainly try implement these on our OJS 2.4.3. You have a good example to call the code for testing?

Regards
Jannie

Hi @makouvlei,

You can click around the OAI interface thanks to the built-in XSL that transofms its XML interface into XHTML; just point your browser at the OAI URL.

Regards,
Alec Smecher
Public Knowledge Project Team

Thanks Alec, I have implemented the modified sql to our OJS and tested the OAI interface pages, all seems to be working and running significantly faster. There is still one reference to the mutex table in the “recordExists” function in lib\pkp\classes\oai\PKPOAIDAO.inc.php. You have a suggestion how to call the new sql for this function? Once that is solved I can remove the mutex table altogether, right?

Regards
Jannie

Hi @makouvlei,

In the pull requests, I changed that function to make use of getRecord instead of implementing its own SQL. See pkp-lib/PKPOAIDAO.inc.php at 005e3d6396821a4e2262e98b84b337ddc8a83a7e · asmecher/pkp-lib · GitHub for details.

Regards,
Alec Smecher
Public Knowledge Project Team

Hi @makouvlei Jannie,

We are dealing with the same problem of slow query joining none empty tombstone table of OJS 2.4.6. I wonder if you can share your implementation code, it will be very helpful to us.

Thank you very much.

Liang

Hi Liang, I pretty much followed the instructions provided by Alec in his replies from April1 and April 3. I’ve included the corrected code below for the affected functions in the 2 files PKPOAIDAO.inc.php & OAIDAO.inc.php, I didn’t share entire files as we have quite a bit of custom code, in our OJS.

Hope this helps.
\lib\pkp\classes\oai\PKPOAIDAO.inc.php

function getEarliestDatestamp($setIds = array()) {
    $result = $this->_getRecordsRecordSet($setIds, null, null, null, null, 'last_modified ASC');
    if ($result->RecordCount() != 0) {
      $row = $result->GetRowAssoc(false);
      $returner = $this->_returnRecordFromRow($row);
    } else $returner = null;
    $result->Close();
    unset($result);
    return OAIUtils::UTCtoTimestamp($returner->datestamp, false);
  }
   
  function recordExists($dataObjectId, $setIds = array()) {
    return $this->getRecord($dataObjectId, $setIds)?true:false;   
  }
   
  function &getRecord($dataObjectId, $setIds = array()) {
    $params = $this->getOrderedRecordParams($dataObjectId, $setIds);
    $result = $this->_getRecordsRecordSet($setIds, null, null, null, $dataObjectId);
    if ($result->RecordCount() != 0) {
      $row =& $result->GetRowAssoc(false);
      $returner =& $this->_returnRecordFromRow($row);
    } else $record = null;
    $result->Close();
    unset($result);
    return $returner;
  }

\classes\oai\ojs\OAIDAO.inc.php

    function _getRecordsRecordSet($setIds, $from, $until, $set, $submissionId = null, $orderBy = 'journal_id, article_id') {
    $journalId = array_shift($setIds);
    $sectionId = array_shift($setIds);

    $params = array();
    if ($journalId) $params[] = (int) $journalId;
    if ($sectionId) $params[] = (int) $sectionId;
    if ($submissionId) $params[] = (int) $submissionId;
    if ($journalId) $params[] = (int) $journalId;
    if ($sectionId) $params[] = (int) $sectionId;
    if (isset($set)) $params[] = $set;
    if ($submissionId) $params[] = (int) $submissionId;
    $result = $this->retrieve(
      'SELECT  LEAST(a.last_modified, i.last_modified) AS last_modified,
      a.article_id AS article_id,
      j.journal_id AS journal_id,
      s.section_id AS section_id,
      i.issue_id,
      NULL AS tombstone_id,
      NULL AS set_spec,
      NULL AS oai_identifier
      FROM
      published_articles pa
      JOIN articles a ON (a.article_id = pa.article_id)
      JOIN issues i ON (i.issue_id = pa.issue_id)
      JOIN sections s ON (s.section_id = a.section_id)
      JOIN journals j ON (j.journal_id = a.journal_id)
      WHERE  i.published = 1 AND j.enabled = 1 AND a.status <> ' . STATUS_DECLINED . '
      ' . ($journalId?' AND j.journal_id = ?':'') . '
      ' . ($sectionId?' AND s.section_id = ?':'') . '
      ' . ($from?' AND LEAST(a.last_modified, i.last_modified) >= ' . $this->datetimeToDB($from):'') . '
      ' . ($until?' AND GREATEST(a.last_modified, i.last_modified) <= ' . $this->datetimeToDB($until):'') . '
      ' . ($submissionId?' AND a.article_id = ?':'') . '
      UNION
      SELECT  dot.date_deleted AS last_modified,
      dot.data_object_id AS article_id,
      tsoj.assoc_id AS assoc_id,
      tsos.assoc_id AS section_id,
      NULL AS issue_id,
      dot.tombstone_id,
      dot.set_spec,
      dot.oai_identifier
      FROM  data_object_tombstones dot
      JOIN data_object_tombstone_oai_set_objects tsoj ON ' . (isset($journalId) ? '(tsoj.tombstone_id = dot.tombstone_id AND tsoj.assoc_type = ' . ASSOC_TYPE_JOURNAL . ' AND tsoj.assoc_id = ?)' : 'tsoj.assoc_id = null') . '
      JOIN data_object_tombstone_oai_set_objects tsos ON ' . (isset($sectionId) ? '(tsos.tombstone_id = dot.tombstone_id AND tsos.assoc_type = ' . ASSOC_TYPE_SECTION . ' AND tsos.assoc_id = ?)' : 'tsos.assoc_id = null') . '
      WHERE  1=1
      ' . (isset($set)?' AND dot.set_spec = ?':'') . '
      ' . ($from?' AND dot.date_deleted >= ' . $this->datetimeToDB($from):'') . '
      ' . ($until?' AND dot.date_deleted <= ' . $this->datetimeToDB($until):'') . '
      ' . ($submissionId?' AND dot.data_object_id = ?':'') . '
      ORDER BY ' . $orderBy,
      $params
    );

    return $result;
  }

Regards
Jannie

@makouvlei
Hi Jannie,

Thank you for your generously sharing your codes.
I basically made the changes exactly as yours, but when I tried to do the oai-pmh harvesting, like ListRecords or ListIdentifiers, it turns out the deleted record notion of the protocol is not supported anymore. The system just returns the current alive records of a certain journal, but not the tombstoned ones. Is this the way the modified code supposed to handle the requests?

Thanks again.

Liang

HI Liang, glad to be of help. Best you direct your question to Alex though, as I don’t have in-depth understanding of the OAI methods.

Regards
Jannie

Thanks Jannie, will do.

@asmecher,
Hi Alec,
could you help to see my previous question asking about if the modified codes support deleted records of the OAI-PMH protocol?

Liang

Hi @Liang,

I can’t comment much on third-party code – but at a glance, yes, it does appear to retain the joins required for tombstones (i.e. deleted records).

Regards,
Alec Smecher
Public Knowledge Project Team