The One You Missed

When you find 30 PHP webshells on a server, you feel thorough. You find them in uploads directories, in wp-includes, in cache folders, in places WordPress should never allow PHP to run. You delete every one. You harden SSH, configure fail2ban, block the attacker's IP addresses, and walk away thinking the job is done.

Then, three days later, the hosting provider shuts down the server because it is brute-forcing IMAP passwords on 70 external hosts. The attacker came back through the one file you missed.[1]

This happened to me this month. A WordPress site on a shared server got compromised through a plugin vulnerability. The attacker uploaded webshells, gained shell access, and ran an IMAP brute-forcer. I cleaned what I thought was everything. I was wrong. A file called newad.php, created during our cleanup window, was a fully functional backdoor disguised as a "Secure Local Task Runner" with a hardcoded IP allowlist. It let the attacker download and execute Perl scripts at will. Another variant, newadd.php, sat right next to it with a doubled letter in the filename, the kind of thing your eye skips over when you are scanning a directory listing in a hurry.

The second compromise was worse than the first. Not because the attacker did more damage, but because it revealed how many things I had gotten wrong in my initial response. Here are the lessons, in no particular order of embarrassment.

Search everywhere, not just the obvious places. I found webshells in uploads directories, which is where most guides tell you to look. What I missed were stubs in subdomain installations: a radio.php in a wiki subdomain, a wp-login.php sitting in an uploads folder from 2021 pretending to be the real WordPress login page, a content.php buried in a CRM plugin's uploads directory. Seventeen zero-byte PHP stubs were scattered across the server. Individually harmless, but they marked where the attacker had tested write access, and any of them could have been replaced with a full shell at any time.[2]

Block the subnet, not just the IP. I blocked the attacker's IP address after the first incident. The attacker came back from a different IP in the same /24 subnet. The hosting provider for the brute-force targets? Same provider, same subnet. Block the whole range.

Fail2ban with maxretry 100 is not fail2ban. It is a decoration. I had it set to allow 100 failed login attempts before banning an IP. That is not a rate limiter. That is a guest book. The correct number for SSH is 3 to 5. For IMAP, similarly aggressive. I now run SSH at maxretry 3 with a 7-day ban, and a dedicated Dovecot jail that watches for IMAP brute-force patterns.

Password authentication was still on. After the first cleanup, I hardened SSH: disabled password auth, set PermitRootLogin to prohibit-password, reduced MaxAuthTries. But the attacker did not need SSH this time. The webshell was the entry point, and the webshell existed because I missed it during cleanup. Hardening SSH after a web-based compromise is like changing the locks on the front door while the window you forgot to check remains wide open.

Check for files created during your cleanup. This was the most painful lesson. The attacker created newad.php on June 8, while I was actively cleaning the server. Either they had a cron job that re-deployed the shell, or they noticed the other shells disappearing and pushed a fresh one. Either way, checking file creation timestamps around the cleanup window would have caught it. I now know to sort by mtime and look for anything created during or immediately after incident response.

PHP execution in uploads directories should be blocked by default. WordPress sites get compromised through plugins constantly. The CVE database is a revolving door of upload vulnerabilities, authenticated bypasses, and arbitrary file writes in plugins like popup-builder, contact-form-7, and WooCommerce extensions.[3] Adding .htaccess rules to disable PHP execution in wp-content/uploads/ is a 30-second fix that would have prevented every single one of the webshells on this server from executing. I had not done it. Now I have.

Change all passwords after a compromise. Every guide says this. I did not do it after the first incident. The attacker had the partei FTP credentials, and I left them in place. Change them all. System users, database users, API keys, everything. If an attacker had shell access, assume every credential on that server is burned.

The server is still in rescue mode. I have a cleanup plan, a hardening plan, and a very specific list of files to check this time. The hosting provider needs a formal statement before reactivation. I will write one, and it will be thorough, because thorough is the only thing that works when you are cleaning up after someone who is more patient than you are.

The hardest part of incident response is not finding the malware. It is believing, honestly, that you have found all of it. The attacker needs one file. You need to find every file. The asymmetry is the whole game.