As I'm sure you noticed, Xepher.net was down for the first half of Saturday. Someone hacked into the server, and replaced every single file with a name starting with "index" with redirects to a hacked forum in germany. I believe I found and fixed the bug they exploited to gain access (a very obscure heap overflow in the perl compatible regular expression library) but I'm not 100% positive that was how they did it. As such, I'm still a bit jumpy about bringing things back online. As such, PLEASE report anything suspicious to me as soon as you see it.
Now, the damage. As I said, they replaced all those files with redirects, completely destroying whatever info used to be in the file. I wrote a script that went and found all such files, then replaced them from the system backup. Problem is, the system backup is a couple months old, and therefore many websites are gonna be rather anachronistic. If you have local copies of your files, go and replace anything named "index" that might have been changed in the past couple months.
Also, I'm gonna be adding more layers of security to the system. I'll post notes on that as I go, but some of it will be stuff you need to know about. Most noteably, I'm going to install a system that watches for failed logins (bad passwords) and will completely ban an IP address if you get too many failures in a row. As such, if you forget/lose a password, do NOT just keep guessing, or you'll get completely blocked for at least an hour.
If anybody has some suggestions for security measures I could add, suggest them here.
Grrr... this whole thing makes me so angry. Today was a beautiful fall day, but instead of getting out and going fishing, I ended up spending the entire day digging through files and code. Completely ruined my Saturday just because some hackers got bored and wanted to show off.
I hate people.
Go find a book/website on hacking whatever OS/software the server is running, and figure out how exactly you'd go about getting into the site if you were a hacker - and then figure out how to keep whatever vulnerabilities you're able to exploit from being exploited. Alternatively, find someone you trust who knows how to hack and get them to attempt to hack in; have them tell you if they could get in, and if they did, how they did it. Each has it's pros and cons; no offense, but someone new to hacking isn't going to be able to have the ability of a pro, but likewise asking someone to try creates a bit of temptation, although you're more likely to get all the flaws in security worked out.
Otherwise, your plan sounds good. Although I'd reccomend backing up files more often (I do weekly backups of critical files to a secondary hard drive on my computer). That way if anything happens, you aren't completely lost - recently, a group I'm a part of was hacked by an anti-Enterprise (or anti-trek in general) hacker, who deleted everything on the server (TrekUnited.com). Having backups every week can prevent the problems posed if the next person who hacks in isn't as nice.
Heh hacking a trekkie site is more nerdy than being a trekkie
Well the old forums URL still leads to all that jarble as do these.
http://schwarzkreuz.xepher.net/
http://immortallychallenged.xepher.net/
http://mystikskies.xepher.net/
http://reality-check.xepher.net/
http://saphanime.xepher.net/
http://tripout.xepher.net/
http://kahvie.xepher.net/
http://art-tard.xepher.net/
http://foolish-mortal.xepher.net/
http://qchiapetp.xepher.net/
http://chibi-summit.xepher.net/
http://sunflower.xepher.net/
http://garnet.xepher.net/
http://kaspall.xepher.net/
http://byteart.xepher.net/
http://jawc.xepher.net/
http://sanna.xepher.net/
http://stxavier.xepher.net/
http://dustbunnies.xepher.net/
http://houseofwizzards.xepher.net/
http://tea-for-one.xepher.net/
Trekkie... good suggestion about thinking like a hacker, but, no offense intended or taken, I AM a good hacker. I've done all I can do to think about how I would hack in, and then to prevent those methods. This particular machine (orca) has been running for 3 years straight, and as you can probably see, it runs a LOT of services, and a LOT of people have fairly open access to it. I see evidence of hacking attempts almost daily, but this is the first time anyone's compromised the system itself. Still though, I think three years is a pretty good record for security on a system that deals out 10,000,000 files a month. And I have had other people try and hack in, the only one who succeeded got as far as guessing a password on an email-only account I used, but couldn't get farther than that. Anyone else is welcome to try, of course. The more heads thinking at it the better. I ask only two things if you game to try though. First, let me know so I don't panic. And two, don't damage anything major in the process. If you want to put a "hahaha" on the main page or something well... fine, but don't go and crash the machine. :-)
Now, as for more regular backups... Not as easy as it sounds. I keep pretty regular backups of my stuff and the systems stuff, but the problem is there are hundreds of users here, and that backup takes hours to do. I don't have room to keep multiple backups either. So when I back up, it overwrites the old backup. That's fine if everything is good when you do it, but often users don't spot a hack or a corrupted file right away, sometimes it can be weeks until they do. If I backed up every day, then by the time they come to me and ask, the backup itself contains only the corrupted file.
Hmm... Thanks to Gwyn, (who's been checking a lot of the sites) some pages were still "hacked" and redirecting. Looking into I see that 343 files (out of over 5000 originally hacked) were unable to be restored, because they didn't exist (or were named differently) when the backup was made. I've simply removed all those files.
thanks for getting it fixed as fast as you did.
Yeah. Thank you so much Xeph for getting it fixed so quickly. We all appreciate it <3
Eugh, hackers annoy the heck out of me.
Bu-ut, I'm just glad that the site's back. Thank-you so much Xepher.
ugh, so glad I keep my own backups on my computer. Or else I'd be manually rewriting all the index files I have.
Thanks for the heads up, I might not have noticed for another week or so.
Oh yeah... I found a super easy way to get into other people's files, so easy even I could do it. Though would you prefer if I told you about it through e-mail or publicly?
Aye Lei... I had to do the same. It's a damn good thing I just made a backup a few days ago after the changes I made to the site interal structure.
Yeah...I only noticed my front page was affected. I haven't tried going through link by link to see if anything else may have happened...I'll probably spend most of my next day off testing that out.
Lei if you mean you can read other people's stuff, that's normal. Most files are world readable by default, unless the owner changes the permissions. Now if you're not just talking about going up a few folders and looking at stuff, but something more hack-like in nature, then by all means, email me and explain.
And as for affected files, it should ONLY be files with "index" in the name.
The server shut down right as I was uploading a forum. I had to erase it and upload it again. Glad you fixed everything though.
ouch Roxor >_<
Yea Xeph, I was just a little surprised (but that doesn't mean I didn't take advantage of it ^^;) when I noticed more .. folders.
I just didn't feel like looking through all my folders and checking what was up and what wasn't. So I just erased everything and uploaded again ^^;
Xepher thanks for getting the system back as quick as you did!
Frack... there's still an entire botnet attacking the server. I'm starting to wonder if the reason this all looked so odd was that there's more than just one person/group trying to get in. Did I mention I hate people?
you're STILL getting hacked?
Quote from: XepherFrack... there's still an entire botnet attacking the server. I'm starting to wonder if the reason this all looked so odd was that there's more than just one person/group trying to get in. Did I mention I hate people?
Write a file in the server's root directory (where most hackers land after they get in), titled "Xepher's Bank Password". Have it set so that anyone who accesses it has the unfortunate consequence of their computer deciding to shut down on them.
Well... I don't know if it could be done, but it's an interesting form of revenge...
Computer's don't work that way... I wish they did sometimes though.
I'm not still getting "hacked" per-say, as there doesn't seem to be anyone who has access, but there are dozens of computers trying to login continually and trying all sorts of likely names and passwords. In other words, the barbarian hordes are at the gates and pounding against them repeatedly.
Here, take a look at a snippit of the log file.
QuoteSep 11 18:16:10 [sshd] Invalid user gitane from 67.15.28.13
Sep 11 18:16:10 [sshd] Invalid user godeffroy from 67.15.28.13
Sep 11 18:16:10 [sshd] Invalid user gracien from 67.15.28.13
Sep 11 18:16:11 [sshd] Invalid user grant from 67.15.28.13
Sep 11 18:16:11 [sshd] Invalid user granville from 67.15.28.13
Sep 11 18:16:12 [sshd] Invalid user grazieele from 67.15.28.13
Sep 11 18:16:12 [sshd] Invalid user gregoire from 67.15.28.13
Sep 11 18:16:13 [sshd] Invalid user gr351gory from 67.15.28.13
Sep 11 18:16:13 [sshd] Invalid user gucci from 67.15.28.13
Sep 11 18:16:13 [sshd] Invalid user guerin from 67.15.28.13
Sep 11 18:16:14 [sshd] Invalid user guerinet from 67.15.28.13
Sep 11 18:16:14 [sshd] Invalid user guibert from 67.15.28.13
Sep 11 18:16:14 [sshd] Invalid user guilette from 67.15.28.13
Sep 11 18:16:15 [sshd] Invalid user guillaume from 67.15.28.13
Sep 11 18:16:15 [sshd] Invalid user guillemin from 67.15.28.13
Sep 11 18:16:16 [sshd] Invalid user guillemot from 67.15.28.13
Sep 11 18:16:16 [sshd] Invalid user guillot from 67.15.28.13
Sep 11 18:16:20 [sshd] Invalid user guimart from 67.15.28.13
Sep 11 18:16:20 [sshd] Invalid user guiot from 67.15.28.13
Sep 11 18:16:20 [sshd] Invalid user guiote from 67.15.28.13
Sep 11 18:16:21 [sshd] Invalid user gunter from 67.15.28.13
Sep 11 18:16:21 [sshd] Invalid user gustav from 67.15.28.13
Sep 11 18:16:22 [sshd] Invalid user guy from 67.15.28.13
Sep 11 18:16:22 [sshd] Invalid user gwendoline from 67.15.28.13
Sep 11 18:16:22 [sshd] Invalid user gwenna353lle from 67.15.28.13
And that's just in those 12 seconds!
Anyway, this sort of thing happens semi-regularly over the past few years. Usually they don't keep it up this long though. This has been pretty much steady for the past week. I'm working on getting a program installed that will watch for repeated attempts and then ban that IP address for an hour or so. Actually, I'm thinking I might not ban it, but use a nifty tarpit thing... The analogy would be that instead of just not answering the requests, or just hanging up on them, it'd put them on indefinite hold, until THEY hang up. It would tie up their system resources somewhat and slow down any attacks they're also running against other people. Problem is, I have to reboot the system to implement some of the kernel features needed for it, and that always makes me nervous.
For now, I just hope none of ya'll have easy to guess passwords!
Need to change my password... :p
gwenna is close to my name! :p
J'accuse!
Heh, good job on fixing it so fast Xeph. Although I have no idea why anyone would be trying to hack our humble sites :/
*gets creople peepy pimples* Gah. People are so grodie. I really sometimes wonder who can manage to be such an asshat.
Any chance that you could report this system trying to hack you?
Thanks for getting it fixed fast, Xepher! It sucks that people want to do that kind of thing though...we're not Microsoft!
Stupidity knows no bounds
Hiya, I may just be being completely stupid here (and this may not be the right place to post this - should be in technical section?) but I've been having problems getting my homepage back up. I had two saved versions in two different places (both recent), and I have tried uploading both in turn, but I can't actually get them to work. As in, I upload them, but each time I just get an Internal Sever Error message on the net instead of my homepage.
Am I being dumb - in which case, someone please tell me what I'm doing wrong before I nut the screen - or is there something else messing it up? The code seems to be exactly as it has always been.
File permissions? If it's a php file it needs to be set securely. Read the "Help & Info" page section on file permissions.
Quote from: DatabitsAny chance that you could report this system trying to hack you?
I'm not sure if you caught everything I said, but it's being attacked by dozens of systems all over the world. This is a botnet... Most likely a bunch of infected machines whose owners have no idea what's going on. It'd be about as useful as trying to report every machine that ever sends you spam.
But... isn't there a way to ban an ip if they use a wrong username/password 10 times or something like that? of course, not eternally... just for 24 hours or so...
That's what I'm implementing, but it requires features added to the kernel, which is the one, single thing in linux that ever requires a reboot.
Picture it this way, it's like reinstalling windows from 1000 miles away, with no monitor and no mouse. As such, I'm taking my time to make sure I do it right the first time, because once I reboot, if it doesn't come back up and let me log back in, then I have to nicely ask the people hosting it to go fix it. They of course, then ask me nicely to pay them good money for their time. :-)
Argh - I completely forgot about the file permissions! It works fine now, thanks Xepher! And thanks for sorting all this out too... Good luck with the fixing.
Oh, a botnet... BLAH!
QuoteYeah, but it's gotta be sourced from one person... whom should burn and suffer in a firey pit of damnation and fury.... or simple be thrown in prison for LIFE.
Honestly though, hackers... WHY? Why put your skills to such poor use like breaking into systems? Wouldn't it make more sense to develop something really clever that could actually be used and make millions?? No.. people are far too stupid to think about that aspect, they'd rather make others lives miserable for no reason at all.
Okay, I have updated the kernel and rebooted the system. Now that the scary part is over, I feel a bit better. I've just turned on fail2ban, a script that will monitor login attempts and block anyone that fails 5 times. This script is new and in beta, so if you run into problems, let me know.
I thought there was a new kernel patch that allowed you to boot into the new kernel without really rebooting the system?
There was a question on Slashdot not too long ago about how to handle repeated log in attempts like this.. I think a lot of the suggestions were similar to what you've implemented though..
Yes, there is a kernel-exec support, but that's... iffy. You still have to bring down the entire system to do it, it just saves you from having to run back through BIOS. That wasn't my worry. It was more that the new kernel would fail... that problem still happens if you use kexec.
As for blocking repeated logins, that should be up and going now.
More software upgrades... and they changed the entire layout for the webserver configuration. As such, I had to rewrite most of it from scratch to match the new layout. I have, of course, been tweaking bits of this for the past 3 years, so it's possible I missed a few custom pieces here and there. Let me know if anything acts weird or broken.
The .htaccess file that was doign redirects via sub domains on my site isn't working correctly. As a matter of fact, even the direct url on the server seems to be broken. Not exactly sure what's going on, but I'm sure it's not the .htaccess file since I tested it by temporarily renaming it.
Yeah, see the other thread about the overnight glitch. Bottom line, I'm working on it.
Ahh... when I logged into the forum this was the only one highlighted as new. I shall read the other! :)
Hey, Xepher, I just had an idea for preventing someone who's managed to hack the site from doing much damage - is there anything you could write that would prevent a user from changing a large amount of files within a short period of time (giving them an error if they try to edit more files than one would normally be able to edit in a short period of time - like 20 files in a minute, or something). It's just an idea, and I'm sure there's probably some reason why you can't do it (I'm not very familiar with everything that Linux/Unix can do, nor am I familiar with servers in general), just figured I should at least suggest it.
Quote from: trekkie1701cHey, Xepher, I just had an idea for preventing someone who's managed to hack the site from doing much damage - is there anything you could write that would prevent a user from changing a large amount of files within a short period of time (giving them an error if they try to edit more files than one would normally be able to edit in a short period of time - like 20 files in a minute, or something). It's just an idea, and I'm sure there's probably some reason why you can't do it (I'm not very familiar with everything that Linux/Unix can do, nor am I familiar with servers in general), just figured I should at least suggest it.
Ack, I regularly change/upload more than 20 files in one session, because I'm lazy and tend to update in clumps.
That wouldn't be a very good system to implement simply because of multiple factors. First off, as Tangle pointed out, some people update their sites often enough to make a lot of changes to things. This would hinder their efforts for effective updates quickly. Also, how would you apply this in terms of uploaded/created files via scripts? Like forums that allow users to upload avatars, or perhaps a comic strip script that lets you upload image pages or frames?
Also... what if you upload a script and find something needs to be fixed or tweaked?
All in all, I think it would cause more headaches than it's worth.
I know this probably sounds silly, but I wish we had some sort of "shield" to bounce back the bot attacks somehow and redirect it elsewhere.
And yes, people are dorkmeisters... =>.<=
There is a shield.... it's called take the server offline. :P
Other than that, isn't much you can do other than be very carefull on security detail.
*shivers* This reminds me of when I used to be a PHP programmer...I hated and still hate hackers. I remembering doing all that wonderful verification with numbers and letters just to check for a real person at some point, which is really the best that I could do against a program trying to get in. That worked for programs, but not so much real people. Real people are trickier to deal with.
(Makes me almost glad I stopped programming. Almost.)
First off... don't panic! Nothing too bad/dangerous, just an explanation for why everything was slow/down for part of today (10/31.) The entire data center became the target of a massive DoS attack. Seems to have basically been a ping flood, as (when I could actually get into the server) I shut down every service, which should mean NO network traffic at all, yet the card was still getting over a megabit every second. Then I lost even my extremely slow connection. When it came back up, the server no longer responded to pings, but everything else worked fine. I think they're blocking all pings to the datacenter right now to fend off this attack.
People should be beaten with a dull spoon.
or a solidified jello rod
Hmmm...So here got hacked too, then? Looks like several webcomic hosring sites (basically the only ones I ever bother keeping up with) have been hacked this month. At least, several of the ones I've visited have mentioned something like this.
Weird...
I never fully understood why people do crap like that. It gives up programmers a bad name.
Oh yay... another hack! Server was down for about 3 hours. Malacious javascript was added to all index files that was designed to infect windows machines with viruses and spyware. The added code opened a tiny iframe that loaded another site. I ran a script that searched for the added string and removed it from all pages. It should be completely clean, but if you encounter any virus warnings or other "odd" behavior from a xepher.net site, please let me know. The original exploit was a bug in OpenSSL, the supposedly "SECURE socket layer" library. I find it ironic that all the major hacks I've seen recently have all been in security software, like the virus scanner and such.
I hate people.
Hmmm, well I guess as long as you caught it in time it's all good.
Phew, I had been uploading some files...then when I went to check on the results the site was down....
For a few seconds there I was like "I....broke Xepher....."
Heheh... No, you didn't, but someone else sure gave it a good try.
Man, people...today just seems to be the day when I learn (more) about how horrible the human race is. Thanks for fixing it so quickly, Xepher.
There are people out there that are truely stupid, simple as that.
I'm a moron... as I tracked down more hacking, I seem to have killed the webserver, and then I went to bed (it was 6am) without checking it. It was off the whole time I slept. My bad.
Ha! That just proves that it happens to the best of us.
Meh, nobody is perfect Xeph. ;)