500 Internal server errors

[reinder]

Rocr.net is 500-ing out! Seems to be only the dynamic pages are affected.  I'll be checking with my programmer, Mithandir, but in the mean time: have there been any changes to the MySQL/PHP setup? Or the CGI configuration?

Also, I can't get into my email (username/password is not recognised) or SSH, so my ability to do my own part in fixing the problem is severely compromised.

[Xepher]

Someone has hacked your site, exploiting some hole in your scripts. I'm hunting stuff down, but it is 4am here, and I've been drinking. :-) Don't worry, I'll find it.

[Xepher]

Okay, traced it further. Gallery, you've got it installed somewhere I think. "view_album.php" was what they exploited to gain access. I'm not quite sure exactly what they did, but basically it used a hole in that script to start their own process. It downloaded more hacks and ran them. As far as I can tell, they just phoned home reporting their runtime environment (such as OS version and software that they ran under) to a few thousand emails. Doesn't look like anything much was compromised, as this didn't seem to get beyond the automated/worm stage. After forensics, I killed all the running processes and deleted the downloaded files (that I could find.) Keep an eye out for odd stuff (like files) that you didn't install. Effectively, they had the same access/permissions as you yourself do, so they could have harmed/changed anything you can. You say your password isn't working? Try it again. The errors you got were actually due to safeguards I have in place. Everyone has limits on the number of programs they can run at once. The hacks took up all those slots, likely preventing you from even running email or logins. Now that they're dead, it should work again. If not, it's possible the hack(ers) changed your password. Let me know if that's the case.

Oh, and upgrade or disable gallery ASAP.

[reinder]

Oh, and upgrade or disable gallery ASAP.

No problem. I've been very fed up with Gallery for some time and have been sitting on the fence waiting to decide what to do about it. I'll talk to Mithandir about moving the contents to WillowCMS.

As for the email, I can now log in, but the inbox isn't working. THis may have been the result of something that I did - I tinkered with the permissions earlier on, in case I needed to delete that 350 MB mailbox. Then when I decided not to do that just yet, I realised I'd forgotten the original permissions. I think they should be rwrr, but I'm not sure.

Once I get that working again, I'll set it to accept only internal system messages. It's not like I use it at all, normally.

[Xepher]

Hmm... your permissions look correct. So it should be working, though I will say that 333MB of Inbox might take a few minutes to load. What are you trying to log into email with? Have you tried the webmail? The hack was sending tons of mail out, so your mailbox might have filled up with bounces or something strange. If it's a viable option, you say you don't use the Inbox here... so if you want, just delete the file. You'll lose all the messages currently in it of course, but it'll be recreated as soon as a new message comes in. That might be the quickest fix. Otherwise give me details and I'll see what I can do.

[reinder]

Well the spam bounces basically happen all the time. I deleted the inbox and will set up filters when I have some time.

[Xepher]

So everything's good for now, right?

Oh, and the rest of ya'll: Take note about gallery.

[reinder]

Seems to be good, although I had a bit of a worry about the page loading speed on ROCR.net earlier on. I'm also somewhat concerned that the hackers may be back to cover their tracks and do more damage that way. I'll keep my eyes open.

I'm going to need to read up on Berkely DB's export format for moving the gallery contents around...

[Xepher]

Well, rocr.net is loading near-instant for me right now. If it does start to slow down, that can be a sign that you've got processes running that shouldn't be. Lots of exploits work that way, putting a script into an endless loop to hold open a back door. That's usually my first sign that someone's trying to hack the server is when I go to the forums here, and it takes too long.