[OJS-3.4.0-5] (Out of Memory) mysqld got signal 6 ; Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware. Thread pointer: 0x78e76b0

Software Support
,
1

OJS-3.4.0-5
MYSQL- 8.0

Hi @jonasraoni @asmecher I am facing mysqld got signal 6 and out of memory issue. During this issue MYSQL Server restarted and this happened for the 3rd time in the last 1 month. I have searched to fix this but without solution.

Below is the MYSQL log

2024-05-29T07:32:00.230381Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2024-05-29T07:32:00.243298Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.
2024-05-29T07:32:00.243456Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.28) starting as process 48007
2024-05-29T07:32:00.415090Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-05-29T07:32:14.588761Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-05-29T07:32:26.087787Z 0 [System] [MY-010229] [Server] Starting XA crash recovery...
2024-05-29T07:32:26.108891Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
2024-05-29T07:32:33.067714Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2024-05-29T07:32:33.067980Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2024-05-29T07:32:33.257125Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2024-05-29T07:32:33.257329Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.28'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server - GPL.
2024-05-29T10:55:51.915917Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2024-05-29T10:55:51.940973Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.
2024-05-29T10:55:51.941047Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.28) starting as process 1749
2024-05-29T10:55:52.175790Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
10:56:03 UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x78e76b0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f358c0afbf0 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x1fdd6dd]
/usr/sbin/mysqld(print_fatal_signal(int)+0x2f3) [0xeebfc3]
/usr/sbin/mysqld(handle_fatal_signal+0xa5) [0xeec075]
/lib64/libpthread.so.0(+0xf630) [0x7f35a0f51630]
/lib64/libc.so.6(gsignal+0x37) [0x7f359f49b387]
/lib64/libc.so.6(abort+0x148) [0x7f359f49ca78]
/lib64/libstdc++.so.6(__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f359fdaba95]
/lib64/libstdc++.so.6(+0x5ea06) [0x7f359fda9a06]
/lib64/libstdc++.so.6(+0x5ea33) [0x7f359fda9a33]
/lib64/libstdc++.so.6(+0x5ec53) [0x7f359fda9c53]
/usr/sbin/mysqld() [0xc308a6]
/usr/sbin/mysqld() [0xc308e2]
/usr/sbin/mysqld(log_sys_init(unsigned int, unsigned long, unsigned int)+0x307) [0x2161bb7]
/usr/sbin/mysqld(srv_start(bool)+0x191b) [0x222d5db]
/usr/sbin/mysqld() [0x20c1fdd]
/usr/sbin/mysqld(dd::bootstrap::DDSE_dict_init(THD*, dict_init_mode_t, unsigned int)+0x8b) [0x1da180b]
/usr/sbin/mysqld(dd::upgrade_57::do_pre_checks_and_initialize_dd(THD*)+0x1a0) [0x1fb0880]
/usr/sbin/mysqld() [0xfae736]
/usr/sbin/mysqld() [0x24b07c0]
/lib64/libpthread.so.0(+0x7ea5) [0x7f35a0f49ea5]
/lib64/libc.so.6(clone+0x6d) [0x7f359f56396d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): Connection ID (thread ID): 1
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2024-05-29T10:56:06.851649Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2024-05-29T10:56:06.857028Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.
2024-05-29T10:56:06.857093Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.28) starting as process 1813
2024-05-29T10:56:06.935791Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-05-29T10:56:19.943590Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-05-29T10:56:25.178229Z 0 [System] [MY-010229] [Server] Starting XA crash recovery...
2024-05-29T10:56:25.191165Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
2024-05-29T10:56:25.868307Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2024-05-29T10:56:25.868680Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2024-05-29T10:56:26.039202Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2024-05-29T10:56:26.039565Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.28'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server - GPL.
2024-05-29T10:58:35.278453Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2024-05-29T10:58:35.326204Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.
2024-05-29T10:58:35.326309Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.28) starting as process 2381
2024-05-29T10:58:35.577788Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-05-29T10:58:52.691934Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-05-29T10:58:56.875889Z 0 [System] [MY-010229] [Server] Starting XA crash recovery...
2024-05-29T10:58:56.892796Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
2024-05-29T10:58:57.053109Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2024-05-29T10:58:57.053341Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2024-05-29T10:58:57.221907Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2024-05-29T10:58:57.222074Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.28'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server - GPL.

Capture

2

Hi @shantanusingh!

In order to decrease the noise in the log, update your MySQL users to use the caching_sha2_password (e.g. ALTER USER username@host IDENTIFIED WITH caching_sha2_password BY 'new_password';), do the same for any repetitive warning :slight_smile:

About the error, it’s out of our league to get it fixed, I’ve never had this issue, and even the message suggests it must be a MySQL bug or hardware error. You might try to downgrade/upgrade MySQL, move the database/system to another machine or get in touch with the MySQL folks.

Best,
Jonas Raoni

3

Hi @shantanusingh,

Just one more suggestion in addition to @jonasraoni’s excellent advice: If you encounter this error during a heavy server load, it might be worth checking your server access log to see if you’re getting heavy activity on a particular URL, e.g. perhaps a bot is indexing your site. If that’s putting heavy load on the server, then it might have caused MySQL to encounter an out-of-memory situation when really the problem is arising somewhere else.

Regards,
Alec Smecher
Public Knowledge Project Team

4

Hi @asmecher @jonasraoni Thanks for your reply.

caching_sha2_password will check, the server is down right now. I think bots are responsible for this, this issue is occurring during heavy access activity.

5

I’ve switched from mysql_native_password to caching_sha2_password and increased the RAM from 47GB to 52GB. Let’s see how it works.

6

We’ve had some incidents with heavy server load, but I didn’t see this error, the server just became unresponsive, until the HTTP requests decreased, the database didn’t go down.

To fight against the bots, you can setup your Nginx/Apache with a module to limit excessive requests, perhaps:

  • Punish/delay excessive requests coming from the same IP
  • Setup a maximum number of requests (the maximum the server can handle without causing overload)

And you can automatically block requests coming from bad bots with something like GitHub - mitchellkrogza/nginx-ultimate-bad-bot-blocker: Nginx Block Bad Bots, Spam Referrer Blocker, Vulnerability Scanners, User-Agents, Malware, Adware, Ransomware, Malicious Sites, with anti-DDOS, Wordpress Theme Detector Blocking and Fail2Ban Jail for Repeat Offenders

Best,
Jonas Raoni

7

@jonasraoni Thanks for the suggestion, I will implement the above suggestions in the new server. We are planning to shift to SSD servers.

1 Like
8

Hi @jonasraoni, can we use restricted crawling in the robot.txt file as mentioned below?
We want to allow Googlebot, Bingbot, etc.. If you suggest/guide some lists of good boats for the Journal we will add them.

User-agent: googlebot
User-agent: google
User-agent: bingbot
User-agent: bing
Disallow: /cache/
9

Hi!

You can take a look at this

I think it’s also good to add some generic protection (drop/slow down connections if too many requests are coming from the same IP or if the server is already overloaded).

Best,
Jonas Raoni

10

Hi @jonasraoni thanks for your valuable suggetion.

I have identified some BadBot and applied some conditions in htaccess file with the help of the above suggestion and now it is working. I have observed for the last three days that BadBot is not coming and server resource usage is normal after implementation.

# BadBot blocking rule 
RewriteCond %{HTTP_USER_AGENT} (SeekportBot|AhrefsBot|Bytespider|Dotbot|Petalbot|DataForSeoBot|SemrushBot) [NC] 
RewriteRule ^(.*)$ – [F,L]