Why MySQL keeps getting killed on your VPS

The database did nothing wrong. It was the biggest program on a server that ran out of memory.

MySQL gets killed when your server runs out of memory. Linux stops one program to keep the rest running, and picks the one using the most memory. On most small servers that is the database. The fix is to find what used up the memory, not to change MySQL.

What happened

Linux has a last-resort guard called the out-of-memory killer. When memory and swap are both used up, it stops the program holding the most memory, at once.

MySQL and MariaDB keep a large cache in memory on purpose, so they usually go first. The site shows a database connection error, systemd often restarts the database a few seconds later, and the cause stays hidden until next time.

Confirm it was memory

Run this on the server:

sudo journalctl -k | grep -i -E "out of memory|oom-kill"

A line like this one is the answer:

Out of memory: Killed process 1234 (mysqld) total-vm:1702380kB, anon-rss:412220kB, ...

The name in brackets is the program that was stopped. It may say mariadbd instead.

journalctl -k shows the current boot. If the server restarted since, add -b -1. Without a saved journal, try sudo dmesg -T | grep -i oom.

No line at all means memory was not the cause. Look in the database’s own error log, usually /var/log/mysql/error.log.

Find what grew

The database was the victim. Something else usually pushed the server over the edge:

To see the biggest programs right now:

ps -eo rss,comm --sort=-rss | head -15

The first column is in kilobytes. Just above the kill line, the kernel lists every program and its size at that moment. Read them with:

sudo journalctl -k | grep -B 60 -i "killed process"

Add -b -1 after -k if the server has restarted since.

Stop it happening again

Work down this list, in order.

  1. Lower pm.max_children so the PHP workers fit: your memory, minus the database, the system and a few hundred megabytes spare, divided by the size of one worker.
  2. Bring the database cache back to a size the server can afford. On 1 GB, the default 128 MB is a sensible start.
  3. Move heavy jobs to the quietest hour, and never run two together.
  4. Add a small swap file if there is none (swapon --show prints nothing). It absorbs short peaks, but slows a server that needs it all the time.
  5. Upgrade memory if the server still runs short on ordinary days. Plan sizes usually double: 1 GB to 2 GB.

Telling Linux never to pick the database is not a fix. Memory is still short, and the next victim may be worse.

If you are deciding how much memory the server should have, our guide to how much RAM a WordPress server needs walks through the sum.

Know the moment it happens

A database that restarts on its own hides the problem until a customer tells you. PimiPulse reads the kernel’s count of programs stopped for lack of memory every minute, and the first stop raises a red warning.

On a 1 GB server, that warning says: “Your server just ran out of memory and had to force-close your database to survive.” And then: “Upgrade memory from your hosting panel (the next common size up from 1 GB is 2 GB), or ask your developer what grew.”

This post is about PimiPulse, part of Pimi.