Upgrade from ojs 2.2.4.0 to 2.4.8.4 - DB table does not exist

Hi,

we are working on test upgrade from ojs 2.2.4.0 to 2.4.8.4 (should later be
followed by upgrade to ojs 3). The run of upgrade script ends up with error

[pre-install]
[load: upgrade.xml]
[version: 2.4.8.4]
[data: dbscripts/xml/upgrade/2.3.0_usersettings.xml]
[schema: lib/pkp/xml/schema/signoff.xml]
[data: dbscripts/xml/upgrade/2.3.0_preupdate.xml]
[data: dbscripts/xml/upgrade/2.3.0_subscription_ip.xml]
[data: dbscripts/xml/upgrade/2.3.3_preupdate1.xml]
[data: dbscripts/xml/upgrade/2.3.3_preupdate2.xml]
[data: dbscripts/xml/upgrade/2.3.3_preupdate3.xml]
[data: dbscripts/xml/upgrade/2.3.3_preaffiliations1.xml]
[data: dbscripts/xml/upgrade/2.3.3_preaffiliations2.xml]
[data: plugins/reports/counter/counter_monthly_log_1_1.xml]
ERROR: Upgrade failed: DB: Table ‘dbname.counter_monthly_log’ doesn’t exist

In reality, the table counter_monthly_log (though empty) does exist in original
DB dbname and partly modified DB dbname - result of unfinished upgrade process
contains the same table, renamed to counter_monthly_log_old.

php errors logfile contains 70 warnings of the type ‘ojs2 has produced an error’,
generated before the start of [pre-install] , but nothing concerning the above error.

Server info:
OS: Ubuntu 18.04
PHP Version: 7.2.17-0ubuntu0.18.04.1
Apache Version: Apache/2.4.29 (Ubuntu)
DB Driver: mysqli

Any suggestions ?

Thanks

Hello,

To present more data concerning our problem, we repeated the upgrade process,
this time setting the parameter ‘debug’ in the configration file config.inc.php
to ‘On’.

Below is the most interesting part of the console output:

[pre-install]
-----
(mysqli): SELECT * FROM versions WHERE current = 1
-----
[load: upgrade.xml]
[version: 2.4.8.4]
[data: dbscripts/xml/upgrade/2.3.0_usersettings.xml]
-----
(mysqli): SHOW TABLES
-----
-----
(mysqli): select version()
-----
-----
(mysqli): SHOW COLUMNS FROM user_settings
-----
-----
(mysqli): ALTER TABLE user_settings CHANGE COLUMN journal_id assoc_id BIGINT(20) NOT NULL DEFAULT 0
-----
[schema: lib/pkp/xml/schema/signoff.xml]
...
[data: dbscripts/xml/upgrade/2.3.0_preupdate.xml]
...
[data: dbscripts/xml/upgrade/2.3.0_subscription_ip.xml]
...
[data: dbscripts/xml/upgrade/2.3.3_preupdate1.xml]
...
[data: dbscripts/xml/upgrade/2.3.3_preupdate2.xml]
...
[data: dbscripts/xml/upgrade/2.3.3_preupdate3.xml]
...
[data: dbscripts/xml/upgrade/2.3.3_preaffiliations1.xml]
...
[data: dbscripts/xml/upgrade/2.3.3_preaffiliations2.xml]
...
[data: plugins/reports/counter/counter_monthly_log_1_1.xml]
-----
(mysqli): SHOW TABLES
-----
-----
(mysqli):
ALTER TABLE counter_monthly_log RENAME TO counter_monthly_log_old

-----
-----
(mysqli):
INSERT INTO counter_monthly_log (year,month,journal_id,count_html,count_pdf,count_other) SELECT year, 1 as month, journal_id, 0, count_jan as count, 0 FROM counter_monthly_log_old UNION SELECT year, 2 as month, journal_id, 0, count_feb as count, 0 FROM counter_monthly_log_old UNION SELECT year, 3 as month, journal_id, 0, count_mar as count, 0 FROM counter_monthly_log_old UNION SELECT year, 4 as month, journal_id, 0, count_apr as count, 0 FROM counter_monthly_log_old UNION SELECT year, 5 as month, journal_id, 0, count_may as count, 0 FROM counter_monthly_log_old UNION SELECT year, 6 as month, journal_id, 0, count_jun as count, 0 FROM counter_monthly_log_old UNION SELECT year, 7 as month, journal_id, 0, count_jul as count, 0 FROM counter_monthly_log_old UNION SELECT year, 8 as month, journal_id, 0, count_aug as count, 0 FROM counter_monthly_log_old UNION SELECT year, 9 as month, journal_id, 0, count_sep as count, 0 FROM counter_monthly_log_old UNION SELECT year, 10 as month, journal_id, 0, count_oct as count, 0 FROM counter_monthly_log_old UNION SELECT year, 11 as month, journal_id, 0, count_nov as count, 0 FROM counter_monthly_log_old UNION SELECT year, 12 as month, journal_id, 0, count_dec as count, 0 FROM counter_monthly_log_old ORDER BY journal_id, year, month

-----
Query:
INSERT INTO counter_monthly_log (year,month,journal_id,count_html,count_pdf,count_other) SELECT year, 1 as month, journal_id, 0, count_jan as count, 0 FROM counter_monthly_log_old UNION SELECT year, 2 as month, journal_id, 0, count_feb as count, 0 FROM counter_monthly_log_old UNION SELECT year, 3 as month, journal_id, 0, count_mar as count, 0 FROM counter_monthly_log_old UNION SELECT year, 4 as month, journal_id, 0, count_apr as count, 0 FROM counter_monthly_log_old UNION SELECT year, 5 as month, journal_id, 0, count_may as count, 0 FROM counter_monthly_log_old UNION SELECT year, 6 as month, journal_id, 0, count_jun as count, 0 FROM counter_monthly_log_old UNION SELECT year, 7 as month, journal_id, 0, count_jul as count, 0 FROM counter_monthly_log_old UNION SELECT year, 8 as month, journal_id, 0, count_aug as count, 0 FROM counter_monthly_log_old UNION SELECT year, 9 as month, journal_id, 0, count_sep as count, 0 FROM counter_monthly_log_old UNION SELECT year, 10 as month, journal_id, 0, count_oct as count, 0 FROM counter_monthly_log_old UNION SELECT year, 11 as month, journal_id, 0, count_nov as count, 0 FROM counter_monthly_log_old UNION SELECT year, 12 as month, journal_id, 0, count_dec as count, 0 FROM counter_monthly_log_old ORDER BY journal_id, year, month
 failed. Table 'dbname.counter_monthly_log' doesn't exist
1146: Table 'dbname.counter_monthly_log' doesn't exist
ADOConnection._Execute(
INSERT INTO counter_monthly_log (year,month,journal_id,count_html,count_pdf,count_other) SELECT year, 1 as month, journal_id..., false)% line  864, file: .../ojs/lib/pkp/lib/adodb/adodb.inc.php
ADOConnection.Execute(
INSERT INTO counter_monthly_log (year,month,journal_id,count_html,count_pdf,count_other) SELECT year, 1 as month, journal_id...)% line  452, file: .../ojs/lib/pkp/classes/install/Installer.inc.php
Installer.executeSQL(
INSERT INTO counter_monthly_log (year,month,journal_id,count_html,count_pdf,count_other) SELECT year, 1 as month, journal_id...)% line  447, file: .../ojs/lib/pkp/classes/install/Installer.inc.php
Installer.executeSQL(Array[2])% line  400, file: .../ojs/lib/pkp/classes/install/Installer.inc.php
Installer.executeAction(Array[3])% line  269, file: .../ojs/lib/pkp/classes/install/Installer.inc.php
ERROR: Upgrade failed: DB: Table 'dbname.counter_monthly_log' doesn't exist

The last 2 mysqli commands are

ALTER TABLE counter_monthly_log RENAME TO counter_monthly_log_old

INSERT INTO counter_monthly_log (year,month,journal_id,count_html,count_pdf,count_other) SELECT year, 1 as month, journal_id, 0, count_jan as count, 0 FROM counter_monthly_log_old UNION SELECT year, 2 as month, journal_id, 0, count_feb as count, 0 FROM counter_monthly_log_old UNION SELECT year, 3 as month, journal_id, 0, count_mar as count, 0 FROM counter_monthly_log_old UNION SELECT year, 4 as month, journal_id, 0, count_apr as count, 0 FROM counter_monthly_log_old UNION SELECT year, 5 as month, journal_id, 0, count_may as count, 0 FROM counter_monthly_log_old UNION SELECT year, 6 as month, journal_id, 0, count_jun as count, 0 FROM counter_monthly_log_old UNION SELECT year, 7 as month, journal_id, 0, count_jul as count, 0 FROM counter_monthly_log_old UNION SELECT year, 8 as month, journal_id, 0, count_aug as count, 0 FROM counter_monthly_log_old UNION SELECT year, 9 as month, journal_id, 0, count_sep as count, 0 FROM counter_monthly_log_old UNION SELECT year, 10 as month, journal_id, 0, count_oct as count, 0 FROM counter_monthly_log_old UNION SELECT year, 11 as month, journal_id, 0, count_nov as count, 0 FROM counter_monthly_log_old UNION SELECT year, 12 as month, journal_id, 0, count_dec as count, 0 FROM counter_monthly_log_old ORDER BY journal_id, year, month

The last one tries to transform the contents of the table counter_monthly_log
(now in table counter_monthly_log_old) and put it into a table of the same name.
Unfortunately, the problem here is that in the moment the table counter_monthly_log
does not exist anymore (as it was renamed to counter_monthly_log_old), so it is not
possible to insert there any data. The problem would not exist if the last command
would have been preceded by a command creating the table counter_monthly_log,
something like

CREATE TABLE `counter_monthly_log` (
  `year` bigint(20) NOT NULL,
  `month` bigint(20) NOT NULL,
  `journal_id` bigint(20) NOT NULL,
  `count_html` bigint(20) NOT NULL DEFAULT '0',
  `count_pdf` bigint(20) NOT NULL DEFAULT '0',
  `count_other` bigint(20) NOT NULL DEFAULT '0',
  ...
);

But it is not the case and so the upgrade process finally ends with an error

1146: Table ‘dbname.counter_monthly_log’ doesn’t exist

Thanks

Hello,

we made another upgrade atempt, this time trying to upgrade from ojs 2.2.4.0 to 2.4.8.5
(as ojs 2.4.8.5 appeared lately to become the current stable ojs 2.x release).
The result was exactly the same as before, i.e.

[pre-install]
[load: upgrade.xml]
[version: 2.4.8.5]
[data: dbscripts/xml/upgrade/2.3.0_usersettings.xml]
[schema: lib/pkp/xml/schema/signoff.xml]
[data: dbscripts/xml/upgrade/2.3.0_preupdate.xml]
[data: dbscripts/xml/upgrade/2.3.0_subscription_ip.xml]
[data: dbscripts/xml/upgrade/2.3.3_preupdate1.xml]
[data: dbscripts/xml/upgrade/2.3.3_preupdate2.xml]
[data: dbscripts/xml/upgrade/2.3.3_preupdate3.xml]
[data: dbscripts/xml/upgrade/2.3.3_preaffiliations1.xml]
[data: dbscripts/xml/upgrade/2.3.3_preaffiliations2.xml]
[data: plugins/reports/counter/counter_monthly_log_1_1.xml]
ERROR: Upgrade failed: DB: Table 'dbname.counter_monthly_log' doesn't exist

which is not surprising, regarding the small amount of changes made from ojs 2.4.8.4.
to 2.4.8.5. (we replaced calls of depricated function split already in ojs 2.4.8.4,
as recommended in the pkp forum).

So, we still don’t have any idea what to do next.

Thanks

Hi @cai_pkjs,

Are you using Counter stats? If not, the quickest way around this is probably to comment out or remove the failing part of the upgrade script. This is in dbscripts/xml/upgrade.xml: ojs/upgrade.xml at ojs-2_4_8-5 · pkp/ojs · GitHub

Regards,
Alec Smecher
Public Knowledge Project Team

Hello Alec,

many thanks for your advise. To follow it, we commented out 4 lines of the script file

dbscripts/xml/upgrade.xml

and started the upgrade process.

The result of our upgrade attempt can briefly be summarized as follows:

[pre-install]
[load: upgrade.xml]
[version: 2.4.8.5]
[data: dbscripts/xml/upgrade/2.3.0_usersettings.xml]
[schema: lib/pkp/xml/schema/signoff.xml]
[data: dbscripts/xml/upgrade/2.3.0_preupdate.xml]
[data: dbscripts/xml/upgrade/2.3.0_subscription_ip.xml]
[data: dbscripts/xml/upgrade/2.3.3_preupdate1.xml]
[data: dbscripts/xml/upgrade/2.3.3_preupdate2.xml]
[data: dbscripts/xml/upgrade/2.3.3_preupdate3.xml]
[data: dbscripts/xml/upgrade/2.3.3_preaffiliations1.xml]
[data: dbscripts/xml/upgrade/2.3.3_preaffiliations2.xml]
[post-install]

...

-----
(mysqli): INSERT INTO controlled_vocabs (assoc_type, assoc_id, controlled_vocab_id, symbolic) VALUES (0, 0, 200, 'nlm30-publication-types')
-----
Query: INSERT INTO controlled_vocabs (assoc_type, assoc_id, controlled_vocab_id, symbolic) VALUES (0, 0, 200, 'nlm30-publication-types') failed. Table 'dbname.controlled_vocabs' doesn't exist
1146: Table 'dbname.controlled_vocabs' doesn't exist
ADOConnection._Execute(INSERT INTO controlled_vocabs (assoc_type, assoc_id, controlled_vocab_id, symbolic) VALUES (0, 0, 200, 'nlm30-publication-types'..., false)% line  864, file: .../ojs/lib/pkp/lib/adodb/adodb.inc.php
ADOConnection.Execute(INSERT INTO controlled_vocabs (assoc_type, assoc_id, controlled_vocab_id, symbolic) VALUES (0, 0, 200, 'nlm30-publication-types'...)% line  452, file: .../ojs/lib/pkp/classes/install/Installer.inc.php
Installer.executeSQL(INSERT INTO controlled_vocabs (assoc_type, assoc_id, controlled_vocab_id, symbolic) VALUES (0, 0, 200, 'nlm30-publication-types'...)% line  447, file: .../ojs/lib/pkp/classes/install/Installer.inc.php
Installer.executeSQL(Array[9])% line  500, file: .../ojs/lib/pkp/classes/plugins/PKPPlugin.inc.php
PKPPlugin.installData(Installer::postInstall, Array[2])% line   83, file: .../ojs/lib/pkp/classes/plugins/MetadataPlugin.inc.php
ojs2 has produced an error
  Message: WARNING: Declaration of FilterGroupDAO::getInsertId() should be compatible with DAO::getInsertId($table = '', $id = '', $callHooks = true)
  In file: .../ojs/lib/pkp/classes/filter/FilterGroupDAO.inc.php
  At line: 215
  Stacktrace:
  Server info:
   OS: Linux
   PHP Version: 7.2.17-0ubuntu0.18.04.1
   Apache Version: N/A
   DB Driver: mysqli
-----
(mysqli): SELECT * FROM filter_groups WHERE symbolic = 'citation=>nlm30'
-----
Query: SELECT * FROM filter_groups WHERE symbolic = 'citation=>nlm30' failed. Table 'dbname.filter_groups' doesn't exist
1146: Table 'dbname.filter_groups' doesn't exist
ADOConnection._Execute(SELECT * FROM filter_groups WHERE symbolic = 'citation=>nlm30')% line  845, file: .../ojs/lib/pkp/lib/adodb/adodb.inc.php
ADOConnection.Execute(SELECT * FROM filter_groups WHERE symbolic = 'citation=>nlm30', Array[1])% line   98, file: .../ojs/lib/pkp/classes/db/DAO.inc.php
DAO.retrieve(SELECT * FROM filter_groups WHERE symbolic = ?, citation=>nlm30)% line   88, file: .../ojs/lib/pkp/classes/filter/FilterGroupDAO.inc.php
FilterGroupDAO.getObjectBySymbolic(citation=>nlm30)% line   32, file: .../ojs/lib/pkp/classes/filter/FilterHelper.inc.php
FilterHelper.installFilterGroups(Object:XMLNode)% line  662, file: .../ojs/lib/pkp/classes/install/Installer.inc.php
<h1>DB Error: Table 'dbname.filter_groups' doesn't exist</h1>ojs2: DB Error: Table 'dbname.filter_groups' doesn't exist

The processing of the DB has been this time much longer, dropping 3 tables

copyed_assignments
layouted_assignments
proof_assignments

creating 21 new tables

books_for_review
books_for_review_authors
books_for_review_settings
dataverse_files
dataverse_studies
object_for_review_assignments
object_for_review_persons
object_for_review_settings
objects_for_review
pln_deposit_objects
pln_deposits
referral_settings
referrals
review_object_metadata
review_object_metadata_settings
review_object_type_settings
review_object_types
signoffs
static_page_settings
static_pages
usage_stats_temporary_records

and stopping on problems with table

filter_groups

The empty table counter_monthly_log was, as expected, left untouched.

The DB table (originally with 80 tables) ended up with 98 (80-3+21) tables,
which is much closer to standard ojs 2.4.8.5 DB having 125 tables than
in our previous upgrade attempts.

Have you any idea how can we move any further ?

Thanks,

Peter

Hi @cai_pkjs,

It looks like you abbreviated some of the install log; can you post the whole thing?

Regards,
Alec Smecher
Public Knowledge Project Team

Hello Alec,

yes, we presented only a small part of the install log, as it is rather long,
trying to cover the essential info.

The file appeared to be too long to include it in this post, I will therefore
send you link you can download it from.

Thanks for your assistance,

Peter

Hi @cai_pkjs,

Hmm, I’m having a hard time figuring out why your upgrade script isn’t running the schema creation scripts like it should. I’d suggest trying…

  1. Restore your database to its 2.2.4.0 state from backup – make sure you drop and re-create the database so upgrade cruft doesn’t get left lying around. (This is a normal part of the upgrade process in case of a failure.)

  2. Before running the upgrade script, try manually creating the missing table(s):

    php php tools/dbXMLtoSQL.php -schema execute lib/pkp/xml/schema/metadata.xml
    
  3. Now try running the upgrade script.

Regards,
Alec Smecher
Public Knowledge Project Team

Hello Alec,

as to your first remark, we never started a new upgrade attempt before restoring
the DB to its 2.2.4.0 state from backup (and checking that it was made as expected).

Following your new advise, we started the command

php tools/dbXMLtoSQL.php -schema execute lib/pkp/xml/schema/metadata.xml

after which 7 new empty tables

citation_settings
citations
filter_groups
filter_settings
filters
metadata_description_settings
metadata_descriptions

were added to the original 2.2.4.0 DB.

Then we started the upgrade process.

The result can briefly be summarized as follows:

[post-install]
[pre-install]
[load: upgrade.xml]
[version: 2.4.8.5]
[data: dbscripts/xml/upgrade/2.3.0_usersettings.xml]
[schema: lib/pkp/xml/schema/signoff.xml]
[data: dbscripts/xml/upgrade/2.3.0_preupdate.xml]
[data: dbscripts/xml/upgrade/2.3.0_subscription_ip.xml]
[data: dbscripts/xml/upgrade/2.3.3_preupdate1.xml]
[data: dbscripts/xml/upgrade/2.3.3_preupdate2.xml]
[data: dbscripts/xml/upgrade/2.3.3_preupdate3.xml]
[data: dbscripts/xml/upgrade/2.3.3_preaffiliations1.xml]
[data: dbscripts/xml/upgrade/2.3.3_preaffiliations2.xml]
[post-install]

...

-----
(mysqli): INSERT INTO email_templates_default_data
(email_key, locale, subject, body, description)
VALUES
('PAYPAL_INVESTIGATE_PAYMENT', 'ar_IQ', ' PayPal ', '              PayPal   {$journalName}.          .\n                       \n    PayPal   .\n\n   :\n{$postInfo}\n\n  ( ):\n{$additionalInfo}\n\n :\n{$serverVars}\n', '                 PayPal        .')
-----
-----
(mysqli): SELECT * FROM versions WHERE product_type = 'core' AND product = 'ojs2' ORDER BY date_installed DESC
-----
Query: SELECT * FROM versions WHERE product_type = 'core' AND product = 'ojs2' ORDER BY date_installed DESC failed. Unknown column 'product_type' in 'where clause'
1054: Unknown column 'product_type' in 'where clause'
ADOConnection._Execute(SELECT * FROM versions WHERE product_type = 'core' AND product = 'ojs2' ORDER BY date_installed DESC)% line  845, file: /var/www/html/ojs/lib/pkp/lib/adodb/adodb.inc.php
ADOConnection.Execute(SELECT * FROM versions WHERE product_type = 'core' AND product = 'ojs2' ORDER BY date_installed DESC, Array[1])% line   98, file: /var/www/html/ojs/lib/pkp/classes/db/DAO.inc.php
DAO.retrieve(SELECT * FROM versions WHERE product_type = ? AND product = ? ORDER BY date_installed DESC, Array[2])% line   98, file: /var/www/html/ojs/lib/pkp/classes/site/VersionDAO.inc.php
VersionDAO.getVersionHistory(core, ojs2)% line  148, file: /var/www/html/ojs/lib/pkp/classes/site/VersionDAO.inc.php
VersionDAO.insertVersion(Object:Version)% line  287, file: /var/www/html/ojs/lib/pkp/classes/install/Installer.inc.php
<h1>DB Error: Unknown column 'product_type' in 'where clause'</h1>ojs2: DB Error: Unknown column 'product_type' in 'where clause'

The DB table (originally with 80+7 tables) ended up with 105 (80+7-3+21) tables.

As the log file is even longer than before, I will send you link you can download
it from.

Thanks,

Peter

Hi @cai_pkjs,

This error is showing up because there is no product_type column on the versions table. Again it looks like a database schema descriptor isn’t being synced to the database where it should be – I’m still not sure what could be causing this, but it could lead to a wild goose chase as you try to track down all the various tables and columns that weren’t added.

To simplify the log file, you might want to turn off the debug option. Otherwise every query will be dumped to the output; this can be helpful sometimes but I suspect you’ll get enough info here without it.

You can work around the specific problem by adding the product_type column to versions before running the upgrade script – you can find this column defined in lib/pkp/xml/schema/common.xml.

If you’re interested in debugging at a deeper level, the upgrade script dbscripts/xml/upgrade.xml defines the whole upgrade process. You’ll see <upgrade> clauses defining tasks for various versions, and a series of <schema> elements for syncing database schema files (to add new tables etc). It’s parsed sequentially, i.e. if an <upgrade> element applies to your version, then a <schema> file runs, then another <upgrade> element, they’re run in that order. By my read, it seems as though <schema> elements are not resulting in the schema actually being applied to your database.

Regards,
Alec Smecher
Public Knowledge Project Team

Hello Alec,

thank you for your new advices.

Yes, in a certain sense we already feel quite some time like chasing a wild goose, who knows when will that comes to an end.

A word to log files. We started to work with debug option set, as it gives more insight into the upgrade process. Of course, the log file is then much longer and I agree, mostly does not contain too valuable data. We post here always just the filtered out basic info more or less similar to what is logged after turning off the debug option and then also final section of logged data saved before encountering an error that stops the whole upgrade process. I have provided you access to the log files from our last two upgrade runs after you have specifically asked for it.

In the next, we shall look closer at the files you suggest as being worth of more thorough examination and also try to eliminate the last encountered problem with product_type column on the versions table.

Thanks once again,

Peter