My Server Was Hacked

On June 23, my server on Dreamhost was hacked and 5 websites that I host were compromised with Pharmaceutical spam.  Below is an overly detailed account of how I discovered the intruision, what I did to find the offensive code on the server, and how I gained a little bit of respect for the hacker who paid attention to the details.  I think I’d like to hire this guy.

Honey, Why Are Their Porno Links On My Site

Women always say porno.  At least my wife does, when she’s talking about the spam ads that she managed to find on her blog, Is It Lupus.  First impression, I didn’t see any links at all.  In order to see the spam links, you needed to click on her home link which was written as <a href=”/”></a> in the code.  If you hit the index page itself, the links were gone, hit the Home link, and spam was back.  This was odd behavior and as I started to dig around, I couldn’t find any obvious infractions in her template code.  Her .htaccess seemed intact and her template files seemed clean.  According to the file dates on the server, I didn’t see any newly created files that could have been causing this mess.  Her blog is an ExpressionEngine site, running an older, out of date version.  I began to think that ExpressionEngine had been compromised and started to dig around inside the code.

After an hour or so of going through the code and the EE forums, I found nothing.  I decided to update the EE software to the latest version after a look in the Change Log noted that one of the version bumps fixed a Cross Site Scripting vulnerability. Perhaps this was my culprit. Update installed, and the spam was gone.  I impressed my wife with my l33t haxor skills in fixing her site.  ”My hero”, she said.  I still wasn’t sure why, how when or where, but at least it was fixed.  

Sadly, she spoke too soon.  The very next day, the pharma-links where back.  Mother. Futon.  What is going on?  I tore into her blog code again. This time I noticed a simple include in the EE index.php file calling a file in the lib directory:

@include('lib/set.php')

At first glance,  I looked right over it.  The naming seeming normal enough.  A quick comparison to a virgin EE file and sure enough, this line of code was foreign.  I hopped on over to the lib directory and opened the file.  

.

Well, hello there.  I echo’d the file and discovered a PHP script that called another file on the server and served it up, but ONLY IF the user agent string was a search engine bot.  You clever little man.  I like your thinking. To anyone and everyone viewing the site in a browser the spam was no where to be found.  To Google and all her friends, I had quickly become a Pharmaceutical king.  For some reason, clicking on the <a href=”/”> link caused the code to think I was a bot and therefore displayed the spam.

Well played, hacker kid. You were smart to serve it up that way, you just didn’t nail down your user agent logic properly.  Further examination of this file showed a fake created date of two years prior. My initial glance through the file structure didn’t draw my attention to those files because you had changed the dates. Details my man, details. I need someone who pays attention to the details.

I Am Jack’s Terminal Knight

Now that I know what I’m looking for, I started using grep to search for files containing similar syntax.  Thanks to a Twitter follower and assumingly all around nice guy, Joel Steidl, I had a nice little grep command to search through all the files on my server for any phrase of my choosing.  I started with the syntax from the encoded file:

grep -lr -e 'base64_encode' *

That search turned up nearly ten spam related files.  In four other domains.  None of these were obvious files.  One sat buried five levels deep in a directly cleverly named like the files around it and of course set with a timestamp to a few years back.  

See that pop.php, that was spam buried in the /motherfuton.com/2006/05/ directory. It was clear that this was a person who was taking their time to bury their malicious code.  I like a person who thinks on their own, isn’t lazy about their work.

As I began to investigate these other four domains, I realized that they too were all infected with the same level of spam as they were already showing as such in Google’s search results.  As a note, I was able to quickly check this using Safari’s Develop menu and setting the user agent string to Google Bot:

Googlebot/2.1 (+http://www.googlebot.com/bot.html)

To make things easier, I added Googlebot to my User Agent dropdown in Safari’s Develop menu.  

If you’re interested in doing the same, here is a walk though of how to add User Agent strings to Safari’s Develop menu:

 http://www.macosxhints.com/article.php?story=20100408082655734

So panic starts to set in now as I realize that this was no longer an ExpressionEngine hack, but what seemed to be a complete compromise of my Dreamhost server.  (My back is getting tense just typing that)  I submit a support ticket with Dreamhost and change every relevant password.

[On that note, 1Password is where it’s at people.  I was not paid to say that.  I should have been. I wonder if I could work out a deal. But I already have a paid license, so what kind of deal could I score? Maybe a license for one of these fine readers who may be going through a tragedy of their own, not nearly as bad as mine of course, because this is the WORST THING EVER IN THE UNIVERSE.  Where was I?]

Passwords changed, I start to think of other vulnerabilities.  Much like my wife’s install of ExpressionEngine, I had two instances of MovableType that were well out of date along with a version of Shortstat (remember that ??). The MovableType installs were rarely used any more so the thought of updating them hadn’t really been in the forefront of my mind.  I begin the update process on both of those.  In the meantime I get a response from Dreamhost support.  It was full of the canned do this, do that stuff, but he was kind enough to include a list of files that he deemed as the culprits.  There were several files I hadn’t discovered via my grep searches and a few that were really well, pretty damn slick.  

The reason those files didn’t result in my grep searches was because they all had slightly different syntax, so a grep search for one specific term, like ‘base64_encode’ didn’t turn up the others that didn’t use that PHP function.  Another example would be a file that used the variable name:

$user_auth

while another seemingly identical file used:

$userauth

Granted this may not have been intentional, but if it was, it’s honestly too bad this kid didn’t put his cleverness to use for something other than hacking.  I could use someone the forethought like this.

I removed all the files that Dreamhost found offensive and continued modifying my grep search towards new syntax that I was finding in the each file.

It’s All About The Backdoor

Now, remember how I wiped the spam the first time, and the next day it was back? There was a reason for that.  As kindly pointed out by the Dreamhost support, hackers usually leave themselves a back door to get back in should you happen to find and wipe their offensive code.  What I found was not one or two but four back doors.  Each buried deep on different domains with different syntax but all resulting in the same herculean script:  a password protected (no joke) back door that was just as functional, if not more, as the FTP client I use daily.  

The hacker could navigate the entire breadth of my server, upload files, rename files, and of course delete files, which thankfully didn’t happen to me.  This was one impressive little script and it was everywhere.  I started modifying my grep searches searching for terminology in the backdoor code and found it again and again all over my server.  Again, each was named something unassuming and fitting for it’s location. Nothing was obvious, it was perfect.

The Cleanup

Aside from removing all the files there were still some steps that needed to be done to clear my good name and return my “portfolio” site, noahstokes.com, to it’s glory as the number one result for ‘best front end developer’ in Google.  Using Google’s Webmaster tools, I was able to resubmit each of the hacked sites to Google for reconsideration.  While not a fast process, Google does promise that it goes through every request.  I’m confident that since I’ve removed all instances of the spam, that Google will in fact begin re-indexing these sites.  

Lessons Learned

I learned a lot in the past few days.  Lessons that I’ll take with me moving forward and that hopefully you can take to heart now to prevent something like this happening to yourself.  

• Always keep software up to date, especially if you use popular open source software.  Hackers are constantly digging trying to find holes.
• Be consistent to check your sites in Google.  Thankfully I caught this early, but if hadn’t who knows how long it could have gone on undetected since the code was only being served up to search bots.
• Setup unique users for each domain that you have.  Yes, this is a pain in the butt.  In my case however, one compromise gave this hacker access to so much more than one site.
• Check your server logs often.  Another dull and boring process, but you can see failed attempts at your site being hacked and button down the hatches if necessary.

In Conclusion

I still don’t know exactly how this person got into my server.  I’m pretty confident it was outdated software, but which version on which domain, I don’t know.  I’ve been running this handy search the past few days every hour or so, just to make sure that no new files are being placed on my server:

find . -type f -name '*' -exec ls -lrt --time-style '+%s' {} \; | sort -r +5 -8 | more

That command searches the current directory for all files sorted by date.  A quick way of seeing if any new files were added that weren’t by my doing.

Overall, this was my fault in not keeping my personal software up to date; so I have no one to blame but myself.  Thankfully none of these were client sites, so there was no awkward conversations, and thankfully no files were removed. To the hacker that did this, job well done.  Your code was clever, well hidden and well thought out.  I’d hire you if you ever wanted to earn some real money.