Hope Isn’t a Recovery Plan

Preparing an aging WordPress server to recover when MySQL goes down

I have an older DigitalOcean Droplet that hosts several websites, including a WordPress site for my old church. Every so often, the website would become unavailable, displaying the familiar WordPress message, “Error establishing a database connection.”

The frustrating part was that the server was still running. The website was responding to requests, Apache was functioning, but WordPress couldn’t communicate with MySQL. My usual solution was to log into DigitalOcean and reboot the entire Droplet. A few minutes later, everything would be working again.

That worked, but it wasn’t much of a solution. The website could potentially sit offline for hours before I noticed, and restarting the server interrupted every website hosted on it.

I wanted to fix the immediate problem without necessarily rebuilding the entire environment.

Recognizing the Warning Signs

The Droplet is pretty modest, with a single vCPU, 512 MB of RAM, and 20 GB of disk space. It’s also running a legacy Ubuntu installation with an older version of MySQL, so we’re dealing with some pretty dated infrastructure.

My suspicion was that the server was running out of available memory and MySQL was either crashing or being terminated by the operating system. When MySQL went down, WordPress couldn’t establish a database connection, and the website would return an error even though Apache was still responding.

Rebooting the Droplet would clear memory, restart the services, and bring everything back online. That seemed to support my theory, but I didn’t have the logs to prove it.

Before changing anything, I checked the server’s memory usage and existing swap configuration. It already had a 1 GB swap file, with approximately 142 MB in use. At the time, there was still available memory, so nothing looked particularly concerning.

Of course, checking memory while the website is working doesn’t tell you much about what happened when it wasn’t.

I could increase the Droplet’s resources, but I wanted to understand the problem before throwing more RAM at it. This is an old server hosting relatively small websites, and I wasn’t particularly interested in paying for additional infrastructure if a simpler solution would keep it operating.

Putting a Contingency Plan in Place

My initial idea was pretty straightforward. Why not create a cron job that checks the website every minute, looks for the WordPress database connection error, and restarts the server when it finds it?

The better approach was to restart MySQL rather than the entire Droplet.

I created a Bash script at /usr/local/sbin/wp-watchdog.sh that monitors the public website at https://myoldchurch.com/.

The script uses curl to request the website and examines the response for the specific WordPress database error. If the error appears, it waits 10 seconds and checks again, repeating the process until it has confirmed three consecutive failures.

I didn’t want a temporary network issue, a slow response, or an unrelated HTTP error triggering an unnecessary database restart.

Once the error is confirmed, the script records the current memory usage and MySQL service status before attempting recovery.

The recovery itself is simple.

service mysql restart

After restarting MySQL, the script waits 20 seconds and checks the website again to verify that WordPress is responding normally.

I also added a 30-minute cooldown to prevent the watchdog from repeatedly restarting MySQL if there was a more serious underlying problem.

Since several websites share this server, I wanted the recovery process to be reasonably conservative. Restarting MySQL affects all of them, even if only one website is being monitored.

Finally, I scheduled the script using root’s crontab.

* * * * * /usr/local/sbin/wp-watchdog.sh

Nothing particularly complicated. One Bash script, one cron entry, and a log file.

Run the Drill Before the Emergency

I wasn’t going to intentionally bring down a production database just to test the recovery script.

Instead, I added a temporary condition that made the script behave as though it had detected a WordPress database connection error. I also added a dry-run safeguard that would stop execution immediately before the actual MySQL restart.

This allowed me to test the detection, verification, logging, and recovery decision without interrupting any websites.

The first test worked as expected.

On October 18, the script detected the simulated error, waited through the additional checks, recorded memory usage, and reached the recovery step.

The log showed:

2026-10-18 23:32:24 WARNING: WordPress database connection error detected.
2026-10-18 23:32:44 CONFIRMED: Three consecutive database errors.
2026-10-18 23:32:44 MEMORY SNAPSHOT:
2026-10-18 23:32:44 MYSQL STATUS:
mysql start/running, process 1376
2026-10-18 23:32:44 DRY RUN: Would restart MySQL here.

That told me the script was following the intended logic.

I removed the simulated failure, removed the dry-run safeguard, verified the Bash syntax, and enabled the cron job.

At that point, there wasn’t much else to do but wait.

When the Plan Was Put to the Test

On October 20, I checked the watchdog log and found something interesting.

The script had actually encountered a database failure and performed a recovery without any intervention from me.

2026-10-20 20:12:08 WARNING: WordPress database connection error detected.
2026-10-20 20:12:33 CONFIRMED: Three consecutive database errors.
2026-10-20 20:12:33 MYSQL STATUS:
 * MySQL is stopped.
2026-10-20 20:12:33 ACTION: Restarting MySQL.
 * Stopping MySQL database server mysqld
   ...done.
 * Starting MySQL database server mysqld
   ...done.
2026-10-20 20:13:01 SUCCESS: WordPress responding normally.

The watchdog detected the failure at 8:12:08 PM and confirmed the website was responding normally again at 8:13:01 PM.

That’s 53 seconds from detection to confirmed recovery, without rebooting the Droplet.

More importantly, the log captured the server’s memory conditions at the time.

The server had only 38 MB of memory available after accounting for reclaimable cache, and swap usage had climbed to 532 MB.

For comparison, during our earlier test, the server had approximately 260 MB available and was only using 141 MB of swap.

That was a pretty substantial difference.

The memory readings supported my original theory, but they still didn’t prove exactly what happened. The log showed MySQL was stopped and the server was under memory pressure. It didn’t explain why the database had stopped.

Fortunately, Linux had already recorded that information.

After Action Report: What Actually Failed?

With the time of the failure established, I went back through Ubuntu’s kernel and system logs.

I searched /var/log/kern.log and /var/log/syslog for memory errors, terminated processes, and MySQL events.

sudo grep -iE 'out of memory|oom|killed process|mysql|mysqld' \
/var/log/kern.log /var/log/syslog 2>/dev/null | tail -80

This is where everything came together.

At 8:10:56 PM, about a minute before the watchdog detected the database failure, the kernel recorded:

apache2 invoked oom-killer
Out of memory: Kill process 1376 (mysqld) score 128 or sacrifice child
Killed process 1376 (mysqld)

There it was.

Linux had exhausted available memory and invoked its out-of-memory killer, commonly called the OOM killer.

When Linux reaches a point where it can’t satisfy memory allocation requests, it may terminate a process to recover memory and keep the system alive.

In this case, an Apache process triggered the OOM handling, and the kernel selected MySQL as the process to terminate.

That distinction matters. It doesn’t necessarily mean Apache has a memory leak or is solely responsible for the problem. It means an Apache process encountered the memory allocation failure, while Linux decided MySQL was the process to kill.

The logs also showed that Ubuntu tried to recover automatically.

init: mysql main process (1376) killed by KILL signal
init: mysql main process ended, respawning

MySQL restarted, but the server was still under memory pressure.

At 8:11:28 PM, the same thing happened again.

apache2 invoked oom-killer
Out of memory: Kill process 14772 (mysqld) score 40 or sacrifice child
Killed process 14772 (mysqld)

MySQL was killed a second time, only about 32 seconds after the first event.

Ubuntu attempted additional restarts, but those attempts failed. Eventually, its service manager gave up.

init: mysql respawning too fast, stopped

That final message explained why the website would remain offline until I manually rebooted the server.

The operating system had mechanisms to restart MySQL when it unexpectedly terminated, but repeated failures caused it to stop trying.

Our watchdog picked up where Ubuntu’s automatic recovery had stopped.

At 8:12:33 PM, it restarted MySQL successfully, and by 8:13:01 PM, WordPress was back online.

Recovery Is Not Prevention

We fixed the recovery problem. We haven’t completely fixed the memory problem.

That’s an important distinction.

My original hypothesis was that memory exhaustion was causing MySQL to become unavailable, and the kernel logs confirmed exactly that.

The watchdog also proved that a full server reboot wasn’t necessary. Restarting MySQL was enough to restore WordPress once the immediate memory pressure had subsided.

Now we have a much better understanding of what’s happening.

The next thing I want to examine is Apache’s configuration and the amount of memory its processes consume. With multiple websites running on a 512 MB Droplet, it wouldn’t take much concurrent PHP activity to put the server under pressure.

There may be opportunities to reduce Apache’s worker limits, tune MySQL’s memory allocations, improve caching, or identify specific requests responsible for unusual spikes.

Longer term, the operating system and database need to be migrated to supported versions. The current software is well beyond its normal support lifetime, and keeping it publicly accessible carries risks beyond the memory issue.

But for now, the immediate reliability problem has a working solution.

What I particularly like about this approach is that we didn’t need to rebuild anything to establish what was happening. We started with a reasonable hypothesis, built a small recovery mechanism, tested it safely, and used the logs from an actual incident to validate the solution.

Then we took the information captured by the watchdog, went back into the system logs, and found the events that caused the failure in the first place.

The site recovered automatically in under a minute from detection, and we now know exactly why MySQL was going down.

That’s a much better position to be in than periodically rebooting a server and hoping the problem doesn’t come back.

• Category: Make/Fix Stuff