OJS 3.3.0.15 Server error 500 index corrupt

[mirkh]

Describe the issue or problem
Since a few days we are having problems with PHP fatal errors informing us that the index of different tables is corrupt. At first it seemed to only be the sessions table, but now it is complaining about for example table user_settings and submission_search_object_keywords. We also get Integrity constraint violation: 1062 Duplicate entry for key ‘sessions_pkey’.

The site is still working but some requests give Server error 500.

Yesterday there was 5742 instances of the word “corrupt” in the error.log, and the day before 1140 instances.

This afternoon we dropped the sessions_pkey index and created a new one. There were none of the database index errors until three hours later. Then we got the same error messages about the sessions table.

Steps I took leading up to the issue
We did not do any changes to the system code this past week.

What application are you using?
OJS 3.3.0.15
We have session_check_ip = On

I guess this could be a pure database or hardware issue? But does anyone have any suggestions for what else we could check or do?

[asmecher]

Hi @mirkh,

I think this is somewhere below the level of OJS; perhaps the MySQL daemon died mid-operation or the server lost power or something similar. You might have better luck looking for database integrity checking tools e.g. on StackOverflow.

One way you might check to see whether the structure of your database is correct is to create a new empty installation of the same version of OJS in a fresh database. Then use mysqldump to dump just the schema, not the data; compare that with the same dump of your current installation.

Regards,
Alec Smecher
Public Knowledge Project Team

[mirkh]

Thanks for your reply. Our database says it’s fine. Did a reboot of the server, but the errors appeared soon again. We’ll investigate further.

[mirkh]

Seems our InnoDB index file has become corrupt. Will follow steps to dump, drop and restore database.