Unlike this blog, Speaking of Clouds is hosted at DreamHost. This is not particularly significant: DreamHost has always provided excellent service, and their customer service guys were immediately responsive when I contacted them. However I’m running on a multiuser system, rather than in my own virtual machine or zone, which meant that certain diagnostic and troubleshooting tools weren’t available. I couldn’t restart the Apache process, or compare logs across multiple websites.
The eventual cleanup was relatively straightforward. Ssh in to the host. Take a recursive listing of the entire filespace, so that I could tell what was changed when. Back up everything. Examine logs. Clean up all of the .htaccess files. Change the keys. Log in to the dashboard. Reinstall WordPress 3.4.1. Identify all of the bogus PHP and HTML files (made easier by the atrocious spelling and grammar of the hackers). Change all the passwords. Reinstall all the plugins and themes. Delete (rather than disabling) everything I’m not actually using. And then back everything up. And all the while, I had three terminal windows tailing the relevant log files.
I must say that I would rather been slogging through the mud at Silverstone, though….
UPDATE July 12, 2012:
This story continues to develop. Yesterday I received an email from a
Russian Lithuanian company (evuln.com), advising me that my site appeared to be hacked, and providing a little bit of more-or-less accurate advice on cleaning it up. The email concluded:
If you are not able to fix this “redirect” problem on your own then we will be glad to help you for a reasonable price.
Oddly, the description that they gave of how I was hacked was slightly inaccurate, and so I ssh’d back into speakingofclouds.com to check. Sure enough, it had been hacked again. I cleaned up as before; this time I touched every file in my WordPress subtree, so that any changes would be immediately apparent.
This morning, I logged back in, and found that my .htaccess files had been changed again. This time I was able to match the modification time to the exact HTTP log entries, and this is what I saw:
126.96.36.199 - - [12/Jul/2012:05:45:44 -0700] "POST /wp-content/uploads/.cache_000.php HTTP/1.1" 200 365 "-" "Mozilla/5.0 (Windows NT 5.1; rv:8.0) Gecko/20100101 Firefox/8.0"
So somehow an executable PHP file had been hidden away in my uploads directory, and was being used to inject stuff into my WordPress configuration. I quarantined the file, then looked around to see if this was a known exploit. I only came across one blog reference, here.
It seems like one really obvious security fix for PHP would be to prevent it from executing hidden files. A quick check suggests that this hasn’t been implemented, though.
UPDATE December 11, 2012:
I’ve received several emails from eVuln.com complaining about this blog piece:
I am concerned about your blog post. It influence our online reputation. I’m sorry about our letter, we just wanted to inform you about security issues in your website.
Since I did no more than state the facts, accurately, I’m not sure what they’re complaining about. In the unlikely event that anyone actually reads this piece and cares about what I wrote, I encourage you to visit the eVuln Labs website and draw your own conclusions.