Why MySQL keeps getting killed on your VPS
The database did nothing wrong. It was the biggest program on a server that ran out of memory.
MySQL gets killed when your server runs out of memory. Linux stops one program to keep the rest running, and picks the one using the most memory. On most small servers that is the database. The fix is to find what used up the memory, not to change MySQL.
What happened
Linux has a last-resort guard called the out-of-memory killer. When memory and swap are both used up, it stops the program holding the most memory, at once.
MySQL and MariaDB keep a large cache in memory on purpose, so they usually go first. The site shows a database connection error, systemd often restarts the database a few seconds later, and the cause stays hidden until next time.
Confirm it was memory
Run this on the server:
sudo journalctl -k | grep -i -E "out of memory|oom-kill"
A line like this one is the answer:
Out of memory: Killed process 1234 (mysqld) total-vm:1702380kB, anon-rss:412220kB, ...
The name in brackets is the program that was stopped. It may say mariadbd instead.
journalctl -k shows the current boot. If the server restarted since, add -b -1. Without a saved journal, try sudo dmesg -T | grep -i oom.
No line at all means memory was not the cause. Look in the database’s own error log, usually /var/log/mysql/error.log.
Find what grew
The database was the victim. Something else usually pushed the server over the edge:
- Too many PHP workers. Each takes 40 to 120 MB, and a burst of visits or a crawler starts them all. Check
pm.max_childrenin the PHP-FPM pool file, for example/etc/php/8.3/fpm/pool.d/www.conf. - A big job at the wrong time. An import, backup or image resize can take hundreds of megabytes for a few minutes.
- A database cache set too large.
innodb_buffer_pool_sizedefaults to 128 MB. A tuning guide may have set it to most of the memory. - A leak. Memory climbs every hour and the kills come a day or two apart.
To see the biggest programs right now:
ps -eo rss,comm --sort=-rss | head -15
The first column is in kilobytes. Just above the kill line, the kernel lists every program and its size at that moment. Read them with:
sudo journalctl -k | grep -B 60 -i "killed process"
Add -b -1 after -k if the server has restarted since.
Stop it happening again
Work down this list, in order.
- Lower
pm.max_childrenso the PHP workers fit: your memory, minus the database, the system and a few hundred megabytes spare, divided by the size of one worker. - Bring the database cache back to a size the server can afford. On 1 GB, the default 128 MB is a sensible start.
- Move heavy jobs to the quietest hour, and never run two together.
- Add a small swap file if there is none (
swapon --showprints nothing). It absorbs short peaks, but slows a server that needs it all the time. - Upgrade memory if the server still runs short on ordinary days. Plan sizes usually double: 1 GB to 2 GB.
Telling Linux never to pick the database is not a fix. Memory is still short, and the next victim may be worse.
If you are deciding how much memory the server should have, our guide to how much RAM a WordPress server needs walks through the sum.
Know the moment it happens
A database that restarts on its own hides the problem until a customer tells you. PimiPulse reads the kernel’s count of programs stopped for lack of memory every minute, and the first stop raises a red warning.
On a 1 GB server, that warning says: “Your server just ran out of memory and had to force-close your database to survive.” And then: “Upgrade memory from your hosting panel (the next common size up from 1 GB is 2 GB), or ask your developer what grew.”
This post is about PimiPulse, part of Pimi.