As you've hopefully seen in other announcements, Xepher.net was hacked again last night. After much investigation, it looks quite likely that there was a backdoor left in place after the last hack, and they just used that to get back in again, despite any security enhancements I made in the interim. The problem here is that, while I think I've found and removed all backdoors and such, there's no way to be positive without a complete system wipe and reinstall. Now, if the machine was sitting here in my room, no problem. I'd take it offline for maybe 8 hours and do it. But it's in Chicago, and I'm in Steamboat. Between shipping both ways and the time to sort and test things, it'd probably mean a week of Xepher.net being completely offline. Also, it would cost me a couple hundred dollars in shipping, and I'd really hate my life because it would eat all my free time when I'm not at work.
Here's my current line of thinking. I think the hacker(s) weren't out to destroy the system, but just spread their viruses and such for other purposes. As such, I believe the damage they've actually done is rather limited, as evidence by everything still running and me still having control of the machine. Now, as I said, I can't "trust" the system, but if it's doing what it need to do... that is, serving your websites up properly, well, I think that's good enough... for now. I'm not going to pull and wipe the system just yet. I'll leave it running for a few more months until I have the time and money to redo it properly. What I ask though, is that since it's possible there's still a hacker with access to things, is that everyone make sure to keep backups of your own data as often as you feel is needed. I do keep system-wide backups, but I'm hesitent to run one of those because all I'd be doing is backing up possibly infected files. If the system is totally taken down, then I will possibly be restoring that backup as an emergency measure. But since it will be a couple months old, many of ya'll could lose newer data if you don't have your own backups. Please realize, I'm talking from paranoia here. This is a worst case scenario. I really don't think the hackers are trying to destroy things, just sneakly spread their worm. As such, I don't anticipate any major disaster, but I'd rather everyone was prepared, just in case.
Now, for future plans... I'm moving back to Texas in a few months. My current plan has me taking over some investment (rental) property, and if that goes well, I'll be getting enough income that I won't have to have a full time job just to get by. That will leave me more time to get Xepher.net sorted out properly. What I want to do is build a new server. The current one (Orca) is now several years in service... the drives have 30,000 hours of power-on time. That's 3.5 years at 24/7 and is definitely "mature" and not at all bad lifetime for something built from spare parts in a college dorm. For a new server, I want to actually drop some real cash on it, build with all new parts. I want a much faster server, with MUCH more storage in a raid array this time 'round. Currently, I'm aiming for about an athlon 64 3200+ with something close to a terabyte of storage. If I do that, I plan to seriously redesign the software side of things as well. The current server setup is pretty much as secure as I can make it while still giving everyone the amount of resources/freedom that I do. I thought it was going to be enough, and it was for several years while we were still small enough to be below the radar. It fended off thousands of "drive by" hacking attempts. The one that finally got through looks to be a concerted effort over many days (possibly weeks or months) by a very determined hacker or group. The new server... I plan to run virtual machines this time around. I'm going to have a master/host that's the actual operating system, but below/inside that I will run virtual machines with a completely seperate OS and everything, and those will run all the actual services. What this gives me is the option to actually wipe and rebuild those virtual machines by remote (using the host system.) So if/when another hacker does get though, I can build a second virtual machine, and switch services over to that one, then go back and wipe the infected machine out completely and rebuild it. I hate having to think this paranoid, but it seems to have become neccessary. The truth is that, no matter how good I design a system to be secure, I'm still at the mercy of the software I use. That means that, even if my design is perfect, a bug in something like the mail server could get the whole machine compromised, which is exactly what happend this last time.
While I'm rebuilding things, I'm going to revamp the hosting side of stuff as well. I'm going to try and implement "Phase 3" as I originally called it. For those wondering, that was the idea that was basically limited (non-machine) accounts specifically for comic hosting. It'd be a lot like keenspace... no applications, everyone gets in, but you can't really do much with it OTHER than host a comic. I also plan to add in some of the things I've been meaning to for a while. Mostly convinence things, like a web-based file manager, and some sort of statistics gathering that's a lot more detailed (and interactive) than the current stats system.
Now, I'd like to hear from ya'll on this, what you think. First off, just any general opinions on what I've mentioned. I have a couple of more specific questions though.
1. Do you think waiting to rebuild the system is a good enough option? Does it worry you undully that the server might be compromised in the meantime?
2. When I rebuild, I'm going to try and resecure things even more than now, but I don't want limit the useability too much. How many non-essential services here do you actually use? To clarify, here's the list of things people can or do use here. Please let me know which ones you actually use, which ones you'd like to use (or might in the future,) or ones I forgot or that you'd like to see
Webserver (essential)
SSH (essential... at least for me, shell/command line access)
SCP (secure file upload, used instead of FTP)
FTP (unsecure file upload... I may be getting rid of this anyway)
Email (username@xepher.net)
Webmail
IMAP (email access protocol)
IMAPS (secure version of above)
POP3 (email access protocol)
POP3S (secure version of above)
SMTP (Outgoing mail server)
Email forwarding
Spam filtering
Wildcard email (anything@username.xepher.net goes to your account here.)
Wildcard filtering (possible future addon... would let mail for address1@username.xepher.net get sorted into a different folder than address2@username.xepher.net. Would require use of email access that supports folders, either IMAP or Webmail.)
MySQL (Used for a lot of forums and similar software)
SQLite (similar to MySQL, but much lighter weight and more secure... used by this forum)
PHPMyAdmin (Web-based interface to for MySQL access/editing.)
HTTPS (Secure webserver)
Cron (process scheduling... run programs at certain time)
Virtual Domains (using a domain you purchased, EX: www.missmab.com rather than missmab.xepher.net)
PHP (scripting... used by lots of webscripts/forums/etc, including the XN newsbox)
Other CGI (Perl, python, and other CGI scripts)
Statistics ( http://xepher.net/stats/ and in the future, more advanced stuff)
Web-based file manger/upload (possible addon in the future)
3. Do you think I should prioritze security, or ease of use / convience, and to what degree? For example, would losing FTP and having only SCP be worth the added security? (WinSCP is just as easy as FTP for direct access, but a lot of editors (like dreamweaver) only support FTP for their builtin file upload.) On the other end, would having to do everything via web-interfaces be too restrictive? That is, having to upload and manage everything in a web-app, with no access to shell, FTP, or other such things. (This is the extreme option, but I believe it would've been secure enough to prevent the most recent hacking attempts, as they all relied on shell access.)
4. What other features or services would you like to see added?
5. How do you feel about the quality of service? By this, I mean how fast the server responds, how well it runs scripts, general performance stuff. This is not to rate MY service as an admin... which has been seriously neglectful when it comes to things like reviewing applications. I apologize for that, but what I'm considering is trying to find a new hosting service for the new server. I notice a decent amount of packet loss and wildly varying ping times with the current one. (On the other hand, it's REALLY affordable, and you get what you pay for.)
6. I want ideas here. I have a couple problems I'm trying to solve. First off, I want Xepher.net to grow, and I want to do so with quality, not just by becoming another geocities. Secondly, I'd like to find a way to get some small income for this thing. I'm looking at investing about a thousand bucks in a new server, and hosting is still costing another hundred a month. If I move to a better host, it could be even more. Right now, the only real income is from donations, and 90% of that is from DMFA ( http://missmab.com ) Still, it's been just over three years since I put the server in chicago and I've spent about 3800 bucks out of pocket for this, not counting hardware costs. I don't intend to forgo the free hosting, but I'm thinking options along the lines of maybe running a seperate "commercial" hosting area, perhaps with dedicated comic services like webcomicsnation.com, which wouldn't have to have a newsbox, or share revenue with donations like the free sites do. Another idea is to maybe offer dedicated game servers for things like counter-strike and such. Other options are maybe stick with the shared donation/ad revenue, but just work hard to get some good, quality content here. I mean, a couple more sites like DMFA is all that's needed. Question is, how to attract such sites? I mean, obviously I need to get one top of my game with applications and whatnot... Speaking of..
7. How can I redo the application system? I want some sort of quality "filter" on new members, but at the same time, I feel bad because the current "wait for Xepher the bevelant dictator to nod" method is leaving lots of worthy applications sitting in limbo because I don't have the time I need to take care of them. I was hoping that with them on the forum, there would be lot of peer review. Don't get me wrong, there are a few members that have been doing a lot of work looking at applications and giving good opinions and advice, and I thank them... but I need more than two or three opinions to let me feel comfortable rubber-stamping something. When I rebuild, should I make a system that requires current members to show up and vote on new people from time to time, or is that trying to force a community where there's not one? How else could I redo it? Do I deputize some trusted people to review and approve/deny applications?
I think that's about it for now, and I apologize for the length of this post, but it's stuff I need to sort out (and apologize for.)
Here's my current line of thinking. I think the hacker(s) weren't out to destroy the system, but just spread their viruses and such for other purposes. As such, I believe the damage they've actually done is rather limited, as evidence by everything still running and me still having control of the machine. Now, as I said, I can't "trust" the system, but if it's doing what it need to do... that is, serving your websites up properly, well, I think that's good enough... for now. I'm not going to pull and wipe the system just yet. I'll leave it running for a few more months until I have the time and money to redo it properly. What I ask though, is that since it's possible there's still a hacker with access to things, is that everyone make sure to keep backups of your own data as often as you feel is needed. I do keep system-wide backups, but I'm hesitent to run one of those because all I'd be doing is backing up possibly infected files. If the system is totally taken down, then I will possibly be restoring that backup as an emergency measure. But since it will be a couple months old, many of ya'll could lose newer data if you don't have your own backups. Please realize, I'm talking from paranoia here. This is a worst case scenario. I really don't think the hackers are trying to destroy things, just sneakly spread their worm. As such, I don't anticipate any major disaster, but I'd rather everyone was prepared, just in case.
Now, for future plans... I'm moving back to Texas in a few months. My current plan has me taking over some investment (rental) property, and if that goes well, I'll be getting enough income that I won't have to have a full time job just to get by. That will leave me more time to get Xepher.net sorted out properly. What I want to do is build a new server. The current one (Orca) is now several years in service... the drives have 30,000 hours of power-on time. That's 3.5 years at 24/7 and is definitely "mature" and not at all bad lifetime for something built from spare parts in a college dorm. For a new server, I want to actually drop some real cash on it, build with all new parts. I want a much faster server, with MUCH more storage in a raid array this time 'round. Currently, I'm aiming for about an athlon 64 3200+ with something close to a terabyte of storage. If I do that, I plan to seriously redesign the software side of things as well. The current server setup is pretty much as secure as I can make it while still giving everyone the amount of resources/freedom that I do. I thought it was going to be enough, and it was for several years while we were still small enough to be below the radar. It fended off thousands of "drive by" hacking attempts. The one that finally got through looks to be a concerted effort over many days (possibly weeks or months) by a very determined hacker or group. The new server... I plan to run virtual machines this time around. I'm going to have a master/host that's the actual operating system, but below/inside that I will run virtual machines with a completely seperate OS and everything, and those will run all the actual services. What this gives me is the option to actually wipe and rebuild those virtual machines by remote (using the host system.) So if/when another hacker does get though, I can build a second virtual machine, and switch services over to that one, then go back and wipe the infected machine out completely and rebuild it. I hate having to think this paranoid, but it seems to have become neccessary. The truth is that, no matter how good I design a system to be secure, I'm still at the mercy of the software I use. That means that, even if my design is perfect, a bug in something like the mail server could get the whole machine compromised, which is exactly what happend this last time.
While I'm rebuilding things, I'm going to revamp the hosting side of stuff as well. I'm going to try and implement "Phase 3" as I originally called it. For those wondering, that was the idea that was basically limited (non-machine) accounts specifically for comic hosting. It'd be a lot like keenspace... no applications, everyone gets in, but you can't really do much with it OTHER than host a comic. I also plan to add in some of the things I've been meaning to for a while. Mostly convinence things, like a web-based file manager, and some sort of statistics gathering that's a lot more detailed (and interactive) than the current stats system.
Now, I'd like to hear from ya'll on this, what you think. First off, just any general opinions on what I've mentioned. I have a couple of more specific questions though.
1. Do you think waiting to rebuild the system is a good enough option? Does it worry you undully that the server might be compromised in the meantime?
2. When I rebuild, I'm going to try and resecure things even more than now, but I don't want limit the useability too much. How many non-essential services here do you actually use? To clarify, here's the list of things people can or do use here. Please let me know which ones you actually use, which ones you'd like to use (or might in the future,) or ones I forgot or that you'd like to see
Webserver (essential)
SSH (essential... at least for me, shell/command line access)
SCP (secure file upload, used instead of FTP)
FTP (unsecure file upload... I may be getting rid of this anyway)
Email (username@xepher.net)
Webmail
IMAP (email access protocol)
IMAPS (secure version of above)
POP3 (email access protocol)
POP3S (secure version of above)
SMTP (Outgoing mail server)
Email forwarding
Spam filtering
Wildcard email (anything@username.xepher.net goes to your account here.)
Wildcard filtering (possible future addon... would let mail for address1@username.xepher.net get sorted into a different folder than address2@username.xepher.net. Would require use of email access that supports folders, either IMAP or Webmail.)
MySQL (Used for a lot of forums and similar software)
SQLite (similar to MySQL, but much lighter weight and more secure... used by this forum)
PHPMyAdmin (Web-based interface to for MySQL access/editing.)
HTTPS (Secure webserver)
Cron (process scheduling... run programs at certain time)
Virtual Domains (using a domain you purchased, EX: www.missmab.com rather than missmab.xepher.net)
PHP (scripting... used by lots of webscripts/forums/etc, including the XN newsbox)
Other CGI (Perl, python, and other CGI scripts)
Statistics ( http://xepher.net/stats/ and in the future, more advanced stuff)
Web-based file manger/upload (possible addon in the future)
3. Do you think I should prioritze security, or ease of use / convience, and to what degree? For example, would losing FTP and having only SCP be worth the added security? (WinSCP is just as easy as FTP for direct access, but a lot of editors (like dreamweaver) only support FTP for their builtin file upload.) On the other end, would having to do everything via web-interfaces be too restrictive? That is, having to upload and manage everything in a web-app, with no access to shell, FTP, or other such things. (This is the extreme option, but I believe it would've been secure enough to prevent the most recent hacking attempts, as they all relied on shell access.)
4. What other features or services would you like to see added?
5. How do you feel about the quality of service? By this, I mean how fast the server responds, how well it runs scripts, general performance stuff. This is not to rate MY service as an admin... which has been seriously neglectful when it comes to things like reviewing applications. I apologize for that, but what I'm considering is trying to find a new hosting service for the new server. I notice a decent amount of packet loss and wildly varying ping times with the current one. (On the other hand, it's REALLY affordable, and you get what you pay for.)
6. I want ideas here. I have a couple problems I'm trying to solve. First off, I want Xepher.net to grow, and I want to do so with quality, not just by becoming another geocities. Secondly, I'd like to find a way to get some small income for this thing. I'm looking at investing about a thousand bucks in a new server, and hosting is still costing another hundred a month. If I move to a better host, it could be even more. Right now, the only real income is from donations, and 90% of that is from DMFA ( http://missmab.com ) Still, it's been just over three years since I put the server in chicago and I've spent about 3800 bucks out of pocket for this, not counting hardware costs. I don't intend to forgo the free hosting, but I'm thinking options along the lines of maybe running a seperate "commercial" hosting area, perhaps with dedicated comic services like webcomicsnation.com, which wouldn't have to have a newsbox, or share revenue with donations like the free sites do. Another idea is to maybe offer dedicated game servers for things like counter-strike and such. Other options are maybe stick with the shared donation/ad revenue, but just work hard to get some good, quality content here. I mean, a couple more sites like DMFA is all that's needed. Question is, how to attract such sites? I mean, obviously I need to get one top of my game with applications and whatnot... Speaking of..
7. How can I redo the application system? I want some sort of quality "filter" on new members, but at the same time, I feel bad because the current "wait for Xepher the bevelant dictator to nod" method is leaving lots of worthy applications sitting in limbo because I don't have the time I need to take care of them. I was hoping that with them on the forum, there would be lot of peer review. Don't get me wrong, there are a few members that have been doing a lot of work looking at applications and giving good opinions and advice, and I thank them... but I need more than two or three opinions to let me feel comfortable rubber-stamping something. When I rebuild, should I make a system that requires current members to show up and vote on new people from time to time, or is that trying to force a community where there's not one? How else could I redo it? Do I deputize some trusted people to review and approve/deny applications?
I think that's about it for now, and I apologize for the length of this post, but it's stuff I need to sort out (and apologize for.)
