How to support Tangut characters?

[Vincent_Cheng]

Describe the issue or problem
When the editor trying to input the Tangut characters either to the title or to the abstract. The title or abstract did not been saved and return to previous status.

Steps I took leading up to the issue
For example:

1. Go to ‘Submissions’ → ‘My Queue’
2. Click on ‘View Submission’
3. Go to ‘Publication’
4. In ‘Title & Abstract’
5. Input Tangut characters in ‘Title’ or ‘Subtitle’ or ‘Abstract’, for instance: 𗭊
6. See the Tangut characters did not been saved.
7. See error

What application are you using?
OJS 3.1.2.4

Additional information

1. Before adding Tangut characters into Title.

2. Input Tangut characters(𗭊) into Title and click save.
   

   

3. The Title back to the previous string and Tangut character 𗭊 did not been saved.

[rcgillis]

Hello @Vincent_Cheng,

The version of OJS that you’re using is no longer supported by PKP. I highly recommend that you upgrade to the newest version of OJS, as it is possible that your issue will be resolved by upgrading (although not guaranteed). However, other community members may wish to offer assistance.

Upgrading instructions are available in the PKP Administrator’s Guide and as part of our [Upgrade Guide] (https://docs.pkp.sfu.ca/dev/upgrade-guide/).

Information about the latest version of OJS can be found on the PKP Website

-Roger
PKP Team

[Vincent_Cheng]

@rcgillis

PKP Website

Hi Roger,

Thank you for your prompt reply and suggestions.

I tried using OJS version 3.4.0.4, but unfortunately, the issue still persists.

Below are the error messages I encountered. I’d appreciate any further suggestions you might have.

[Fri Dec 20 03:56:27.106614 2024] [php:notice] [pid 27] [client 10.8.0.6:64820] Slim Application Error:\nType: Illuminate\Database\QueryException\nCode: 22007\nMessage: SQLSTATE[22007]: Invalid datetime format: 1366 Incorrect string value: '\xF0\x97\xAD\x8A \xC2…' for column ojs.publication_settings.setting_value at row 1 (SQL: update publication_settings set setting_value = Revisiting the Tangut Directional Prefix \xf0\x97\xad\x8a ·jij1 where (publication_id = 1 and locale = en and setting_name = title) limit 1)\nFile: /var/www/html/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Database/Connection.php\nLine: 760\nTrace: #0 /var/www/html/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Database/Connection.php(720): Illuminate\Database\Connection->runQueryCallback()\n#1 /var/www/html/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Database/Connection.php(577): Illuminate\Database\Connection->run()\n#2 /var/www/html/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Database/Connection.php(510): Illuminate\Database\Connection->affectingStatement()\n#3 /var/www/html/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(3357): Illuminate\Database\Connection->update()\n#4 /var/www/html/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(3399): Illuminate\Database\Query\Builder->update()\n#5 /var/www/html/lib/pkp/classes/core/EntityDAO.php(225): Illuminate\Database\Query\Builder->updateOrInsert()\n#6 /var/www/html/lib/pkp/classes/publication/DAO.php(239): PKP\core\EntityDAO->_update()\n#7 /var/www/html/lib/pkp/classes/publication/Repository.php(394): PKP\publication\DAO->update()\n#8 /var/www/html/lib/pkp/api/v1/submissions/PKPSubmissionHandler.php(1118): PKP\publication\Repository->edit()\n#9 [internal function]: PKP\API\v1\submissions\PKPSubmissionHandler->editPublication()\n#10 /var/www/html/lib/pkp/lib/vendor/slim/slim/Slim/Handlers/Strategies/RequestResponse.php(40): call_user_func()\n#11 /var/www/html/lib/pkp/lib/vendor/slim/slim/Slim/Route.php(281): Slim\Handlers\Strategies\RequestResponse->__invoke()\n#12 /var/www/html/lib/pkp/lib/vendor/slim/slim/Slim/MiddlewareAwareTrait.php(117): Slim\Route->__invoke()\n#13 /var/www/html/lib/pkp/lib/vendor/slim/slim/Slim/Route.php(268): Slim\Route->callMiddlewareStack()\n#14 /var/www/html/lib/pkp/lib/vendor/slim/slim/Slim/App.php(503): Slim\Route->run()\n#15 /var/www/html/lib/pkp/classes/security/authorization/internal/ApiAuthorizationMiddleware.php(90): Slim\App->__invoke()\n#16 [internal function]: PKP\security\authorization\internal\ApiAuthorizationMiddleware->__invoke()\n#17 /var/www/html/lib/pkp/lib/vendor/slim/slim/Slim/DeferredCallable.php(57): call_user_func_array()\n#18 [internal function]: Slim\DeferredCallable->__invoke()\n#19 /var/www/html/lib/pkp/lib/vendor/slim/slim/Slim/MiddlewareAwareTrait.php(70): call_user_func()\n#20 /var/www/html/lib/pkp/classes/security/authorization/internal/ApiCsrfMiddleware.php(56): Slim\App->Slim\{closure}()\n#21 [internal function]: PKP\security\authorization\internal\ApiCsrfMiddleware->__invoke()\n#22 /var/www/html/lib/pkp/lib/vendor/slim/slim/Slim/DeferredCallable.php(57): call_user_func_array()\n#23 [internal function]: Slim\DeferredCallable->__invoke()\n#24 /var/www/html/lib/pkp/lib/vendor/slim/slim/Slim/MiddlewareAwareTrait.php(70): call_user_func()\n#25 /var/www/html/lib/pkp/classes/security/authorization/internal/ApiTokenDecodingMiddleware.php(140): Slim\App->Slim\{closure}()\n#26 [internal function]: PKP\security\authorization\internal\ApiTokenDecodingMiddleware->__invoke()\n#27 /var/www/html/lib/pkp/lib/vendor/slim/slim/Slim/DeferredCallable.php(57): call_user_func_array()\n#28 [internal function]: Slim\DeferredCallable->__invoke()\n#29 /var/www/html/lib/pkp/lib/vendor/slim/slim/Slim/MiddlewareAwareTrait.php(70): call_user_func()\n#30 /var/www/html/lib/pkp/classes/handler/APIHandler.php(83): Slim\App->Slim\{closure}()\n#31 [internal function]: PKP\handler\APIHandler->PKP\handler\{closure}()\n#32 /var/www/html/lib/pkp/lib/vendor/slim/slim/Slim/DeferredCallable.php(57): call_user_func_array()\n#33 [internal function]: Slim\DeferredCallable->__invoke()\n#34 /var/www/html/lib/pkp/lib/vendor/slim/slim/Slim/MiddlewareAwareTrait.php(70): call_user_func()\n#35 /var/www/html/lib/pkp/classes/handler/APIHandler.php(100): Slim\App->Slim\{closure}()\n#36 [internal function]: PKP\handler\APIHandler->PKP\handler\{closure}()\n#37 /var/www/html/lib/pkp/lib/vendor/slim/slim/Slim/DeferredCallable.php(57): call_user_func_array()\n#38 [internal function]: Slim\DeferredCallable->__invoke()\n#39 /var/www/html/lib/pkp/lib/vendor/slim/slim/Slim/MiddlewareAwareTrait.php(70): call_user_func()\n#40 /var/www/html/lib/pkp/classes/handler/APIHandler.php(105): Slim\App->Slim\{closure}()\n#41 [internal function]: PKP\handler\APIHandler->PKP\handler\{closure}()\n#42 /var/www/html/lib/pkp/lib/vendor/slim/slim/Slim/DeferredCallable.php(57): call_user_func_array()\n#43 [internal function]: Slim\DeferredCallable->__invoke()\n#44 /var/www/html/lib/pkp/lib/vendor/slim/slim/Slim/MiddlewareAwareTrait.php(70): call_user_func()\n#45 /var/www/html/lib/pkp/lib/vendor/slim/slim/Slim/MiddlewareAwareTrait.php(117): Slim\App->Slim\{closure}()\n#46 /var/www/html/lib/pkp/lib/vendor/slim/slim/Slim/App.php(392): Slim\App->callMiddlewareStack()\n#47 /var/www/html/lib/pkp/lib/vendor/slim/slim/Slim/App.php(297): Slim\App->process()\n#48 /var/www/html/lib/pkp/classes/core/APIRouter.php(110): Slim\App->run()\n#49 /var/www/html/lib/pkp/classes/core/Dispatcher.php(165): PKP\core\APIRouter->route()\n#50 /var/www/html/lib/pkp/classes/core/PKPApplication.php(388): PKP\core\Dispatcher->dispatch()\n#51 /var/www/html/index.php(21): PKP\core\PKPApplication->execute()\n#52 {main}\nPrevious error:\nType: PDOException\nCode: 22007\nMessage: SQLSTATE[22007]: Invalid datetime format: 1366 Incorrect string value: '\xF0\x97\xAD\x8A \xC2…' for column ojs.publication_settings.setting_value at row 1\nFile: /var/www/html/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Database/Connection.php\nLine: 570\nTrace: #0 /var/www/html/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Database/Connection.php(570): PDOStatement->execute()\n#1 /var/www/html/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Database/Connection.php(753): Illuminate\Database\Connection->Illuminate\Database\{closure}()\n#2 /var/www/html/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Database/Connection.php(720): Illuminate\Database\Connection->runQueryCallback()\n#3 /var/www/html/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Database/Connection.php(577): Illuminate\Database\Connection->run()\n#4 /var/www/html/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Database/Connection.php(510): Illuminate\Database\Connection->affectingStatement()\n#5 /var/www/html/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(3357): Illuminate\Database\Connection->update()\n#6 /var/www/html/lib/pkp/lib/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(3399): Illuminate\Database\Query\Builder->update()\n#7 /var/www/html/lib/pkp/classes/core/EntityDAO.php(225): Illuminate\Database\Query\Builder->updateOrInsert()\n#8 /var/www/html/lib/pkp/classes/publication/DAO.php(239): PKP\core\EntityDAO->_update()\n#9 /var/www/html/lib/pkp/classes/publication/Repository.php(394): PKP\publication\DAO->update()\n#10 /var/www/html/lib/pkp/api/v1/submissions/PKPSubmissionHandler.php(1118): PKP\publication\Repository->edit()\n#11 [internal function]: PKP\API\v1\submissions\PKPSubmissionHandler->editPublication()\n#12 /var/www/html/lib/pkp/lib/vendor/slim/slim/Slim/Handlers/Strategies/RequestResponse.php(40): call_user_func()\n#13 /var/www/html/lib/pkp/lib/vendor/slim/slim/Slim/Route.php(281): Slim\Handlers\Strategies\RequestResponse->__invoke()\n#14 /var/www/html/lib/pkp/lib/vendor/slim/slim/Slim/MiddlewareAwareTrait.php(117): Slim\Route->__invoke()\n#15 /var/www/html/lib/pkp/lib/vendor/s

Thank you very much!

Best regards,
Vincent

[asmecher]

Hi @Vincent_Cheng,

I think the issue is in MySQL rather than OJS – most MySQL installations will prefer the utf8_general_ci collation when creating content with UTF8, but that’s only 3-byte Unicode, and the characters you’d like to use require 4 bytes.

There’s a good article here on how to convert from utf8_general_ci to utf8mb4:

https://mathiasbynens.be/notes/mysql-utf8mb4

OJS should be neutral to this, but you should also set the collation option in config.inc.php to match.

There are a few other threads on the forum related to this if you search for utf8mb4.

Thanks, and please report back if this helps! And of course don’t forget to take a good backup before modifying the database.

Regards,
Alec Smecher
Public Knowledge Project Team

[Vincent_Cheng]

Hi @asmecher,

Thank you for your prompt reply and detailed suggestions. I’ll give them a try. Much appreciated!

Best regards,
Vincent Cheng