Hacked! (Or "Why I Hate People")

[Xepher]

As I'm sure you noticed, Xepher.net was down for the first half of Saturday. Someone hacked into the server, and replaced every single file with a name starting with "index" with redirects to a hacked forum in germany. I believe I found and fixed the bug they exploited to gain access (a very obscure heap overflow in the perl compatible regular expression library) but I'm not 100% positive that was how they did it. As such, I'm still a bit jumpy about bringing things back online. As such, PLEASE report anything suspicious to me as soon as you see it.

Now, the damage. As I said, they replaced all those files with redirects, completely destroying whatever info used to be in the file. I wrote a script that went and found all such files, then replaced them from the system backup. Problem is, the system backup is a couple months old, and therefore many websites are gonna be rather anachronistic. If you have local copies of your files, go and replace anything named "index" that might have been changed in the past couple months.

Also, I'm gonna be adding more layers of security to the system. I'll post notes on that as I go, but some of it will be stuff you need to know about. Most noteably, I'm going to install a system that watches for failed logins (bad passwords) and will completely ban an IP address if you get too many failures in a row. As such, if you forget/lose a password, do NOT just keep guessing, or you'll get completely blocked for at least an hour.

If anybody has some suggestions for security measures I could add, suggest them here.

Grrr... this whole thing makes me so angry. Today was a beautiful fall day, but instead of getting out and going fishing, I ended up spending the entire day digging through files and code. Completely ruined my Saturday just because some hackers got bored and wanted to show off.

I hate people.

[trekkie1701c]

Go find a book/website on hacking whatever OS/software the server is running, and figure out how exactly you'd go about getting into the site if you were a hacker - and then figure out how to keep whatever vulnerabilities you're able to exploit from being exploited.  Alternatively, find someone you trust who knows how to hack and get them to attempt to hack in; have them tell you if they could get in, and if they did, how they did it.  Each has it's pros and cons; no offense, but someone new to hacking isn't going to be able to have the ability of a pro, but likewise asking someone to try creates a bit of temptation, although you're more likely to get all the flaws in security worked out.

Otherwise, your plan sounds good.  Although I'd reccomend backing up files more often (I do weekly backups of critical files to a secondary hard drive on my computer).  That way if anything happens, you aren't completely lost - recently, a group I'm a part of was hacked by an anti-Enterprise (or anti-trek in general) hacker, who deleted everything on the server (TrekUnited.com).  Having backups every week can prevent the problems posed if the next person who hacks in isn't as nice.

[Gwyn]

Heh hacking a trekkie site is more nerdy than being a trekkie

[Gwyn]

Well the old forums URL still leads to all that jarble as do these.

http://schwarzkreuz.xepher.net/
http://immortallychallenged.xepher.net/
http://mystikskies.xepher.net/
http://reality-check.xepher.net/
http://saphanime.xepher.net/
http://tripout.xepher.net/
http://kahvie.xepher.net/
http://art-tard.xepher.net/
http://foolish-mortal.xepher.net/
http://qchiapetp.xepher.net/
http://chibi-summit.xepher.net/
http://sunflower.xepher.net/
http://garnet.xepher.net/
http://kaspall.xepher.net/
http://byteart.xepher.net/
http://jawc.xepher.net/
http://sanna.xepher.net/
http://stxavier.xepher.net/
http://dustbunnies.xepher.net/
http://houseofwizzards.xepher.net/
http://tea-for-one.xepher.net/

[Xepher]

Trekkie... good suggestion about thinking like a hacker, but, no offense intended or taken, I AM a good hacker. I've done all I can do to think about how I would hack in, and then to prevent those methods. This particular machine (orca) has been running for 3 years straight, and as you can probably see, it runs a LOT of services, and a LOT of people have fairly open access to it. I see evidence of hacking attempts almost daily, but this is the first time anyone's compromised the system itself. Still though, I think three years is a pretty good record for security on a system that deals out 10,000,000 files a month. And I have had other people try and hack in, the only one who succeeded got as far as guessing a password on an email-only account I used, but couldn't get farther than that. Anyone else is welcome to try, of course. The more heads thinking at it the better. I ask only two things if you game to try though. First, let me know so I don't panic. And two, don't damage anything major in the process. If you want to put a "hahaha" on the main page or something well... fine, but don't go and crash the machine. :-)

Now, as for more regular backups... Not as easy as it sounds. I keep pretty regular backups of my stuff and the systems stuff, but the problem is there are hundreds of users here, and that backup takes hours to do. I don't have room to keep multiple backups either. So when I back up, it overwrites the old backup. That's fine if everything is good when you do it, but often users don't spot a hack or a corrupted file right away, sometimes it can be weeks until they do. If I backed up every day, then by the time they come to me and ask, the backup itself contains only the corrupted file.

[Xepher]

Hmm... Thanks to Gwyn, (who's been checking a lot of the sites) some pages were still "hacked" and redirecting. Looking into I see that 343 files (out of over 5000 originally hacked) were unable to be restored, because they didn't exist (or were named differently) when the backup was made. I've simply removed all those files.

[jun]

thanks for getting it fixed as fast as you did.

[Slang]

Yeah. Thank you so much Xeph for getting it fixed so quickly. We all appreciate it <3

[Glennoar]

Eugh, hackers annoy the heck out of me.  

Bu-ut, I'm just glad that the site's back.  Thank-you so much Xepher.

[Lei]

ugh, so glad I keep my own backups on my computer. Or else I'd be manually rewriting all the index files I have.

 Thanks for the heads up, I might not have noticed for another week or so.

 Oh yeah... I found a super easy way to get into other people's files, so easy even I could do it. Though would you prefer if I told you about it through e-mail or publicly?

[Databits]

Aye Lei... I had to do the same. It's a damn good thing I just made a backup a few days ago after the changes I made to the site interal structure.

[DC Bueller]

Yeah...I only noticed my front page was affected.  I haven't tried going through link by link to see if anything else may have happened...I'll probably spend most of my next day off testing that out.

[Xepher]

Lei if you mean you can read other people's stuff, that's normal. Most files are world readable by default, unless the owner changes the permissions. Now if you're not just talking about going up a few folders and looking at stuff, but something more hack-like in nature, then by all means, email me and explain.

And as for affected files, it should ONLY be files with "index" in the name.

[RoxorFuxor]

The server shut down right as I was uploading a forum.  I had to erase it and upload it again.  Glad you fixed everything though.

[Lei]

ouch Roxor >_<

 Yea Xeph, I was just a little surprised (but that doesn't mean I didn't take advantage of it ^^;) when I noticed more .. folders.

 I just didn't feel like looking through all my folders and checking what was up and what wasn't. So I just erased everything and uploaded again ^^;