So I think I've got the parts for the new server picked out. I haven't ordered yet, as I'm just being a little extra cautious to make sure everything on the motherboard I picked out will work with linux properly.
What I'm looking at:
Athena Power RM-2U2015SV40 case (http://www.newegg.com/Product/Product.asp?Item=N82E16811192036) $130 (Yay a real server case!)
Intel MB BOXDG965WHMKR Motherboard (http://www.newegg.com/Product/Product.asp?Item=N82E16813121052R) $72
2GB DDR2 800mhz Ram (http://www.newegg.com/Product/Product.asp?Item=N82E16820231098) $175
E6400 Core 2 Duo CPU (http://www.newegg.com/Product/Product.asp?Item=N82E16819115004) $222
And 4 of these: 320GB Hard Drives (http://www.newegg.com/Product/Product.asp?Item=N82E16822148140) $90x4
Total: $959 + Shipping
I'll also probably get a 2GB compact flash card and an IDE to flash adapter, giving me a solid state recovery "disk" for emergency use. It's about $30 for both. The 4 drives will be in a raid5 array, so total usable storage will be 960GB. The lost space goes to redundancy, so if any one drive fails, no data is lost.
The real question this brings up is how to go about the transfer. It'll take me several weeks to configure at test the new server once I get the parts, but when it comes time to actually replace the old one, I have to figure out the plan. See, it'll take about 3 days to ship the new one up there, and I won't have access to it during that time. That means that any data I have on it from users will be 3 days old by the time it gets there, and everyone will lose any updates they've done in those 3 days once the old server is powered down.
So I have a few options:
#1 I can lock the old server into read-only access when I sync the new one. This would happen right before I ship it to the datacenter. That would leave things sorta running while the new one is in transit, but no updates would be possible, no forum posts, no email, etc. But on the plus side, there would be nothing tricky to do once it got there. They'd just swap it for the old one, and everything would start running normally as soon as that was done.
#2 I can just warn everyone that as of a certain time/date, updates after that will be lost. People can still use forums and such for the intervening days, but everything will just "revert" back about 3 days once the new server comes online. Double trick with this is email. I don't think many people would be willing to lose 3 days of incoming email.
#3 Basically the opposite of #1. Leave the old server running normally, but new server would be read-only until I get the old machine back in my own hands and can get the data off it and send it to the new machine.
#4 Host the new machine somewhere else, and have some overlap time before I cancel the hosting on the old one. This would let me do a live sync, and would be nearly seamless. The downside is that I can't find the sort of bandwidth we need at anywhere near as good of a price. We basically need 3-5mbps connection, or around 1.5 terabyte a month, and space for a 2U box. Right now we've got 10mbps for $109/month I can't seem to beat that with a stick.
Anyway, I'm posting this here both as a heads up on what'll be happening in the next couple months, and also to ask for opinions and/or ideas. I think the server components are pretty much the right choice, but if someone's got a better idea, feel free to suggest things. As for how to do the transfer... I really do need some opinions on that. Also, if anyone can find me a good deal on server colocation that's even close to the price I pay now, I'd definitely like to know.
thanks for the heads up. Just let us know. I'll be looking forward to a viable excuse as to why I can't update for a few days :P
I like (well, not like but appreciate) what DeviantARt does, which is put up a big notice at the top of each page saying "this server is in read-only mode for maintenance".
Anyway, I can cope with any approach as long as I get a heads-up so I can communicate to my readers and point them to a mirror for the duration.
I plan to do something similar, with clear notice being made. Also, I will give plenty of notice before doing anything. Probably 4 weeks general notice, but a minimum of 2 weeks with an exact date. I would like to find another option, but I'm not sure there is one. I can hope that shipping will take less than 3 full days too, as that's merely an estimate.
Hmm... new thought. Maybe I can keep a local mirror on my personal computer as well, and sync that every few hours while the new server is in shipment, then I can sync the new server from my desktop once it's in place. Wouldn't be perfect, but would bring downtime down to maybe 6-12 hours or so.
I don't know anything about server colocation, but option #1 sounds great.
While I like the idea of option 4, we're not trying to break the bank, and option 1 sounds the best.
If you wait a little longer you may be able to get a dual core for a bit cheaper than they are even now. About beginning summer quad cores are being released from AMD, so I'd suspect that most dual core cpu's will probably fall a bit in price.
Yeah, but I've seen intel's roadmap, as well as reviews of the AMD 4x4 chips (which aren't as hot as I'd hoped), while I'm sure prices will drop, it won't be anything dramatic until probably the end of summer. Saving 70 bucks isn't worth that much delay, IMHO. I'll keep my eyes open until I actually buy stuff though, if I hear something new is coming out like next week, I'll give it a tad longer.
Just ordered the server. Upgraded a few things from the list above. Got a larger/better case that cost $180, and added on the compact flash thing I mentioned. Also, the motherboard went back up to full price ($130) so total came in around $1150 for the whole thing. Should be whoppingly overpowered for what we need, but give things room to grow. Once I get the parts next week, I'll put it all together and start building the software side of things. That'll probably take a few weeks. After that, I'll give it probably two weeks of all out stress tests before I'm ready to send it off. I'll keep you guys up to date as I go.
Sounds good! I wish I got more traffic so I could bring some more donations in, but I help with what I can.
No worries. Part of the reason for the overkill is "future plans." I still want to move forward with the Artica Project, though I'm working on a better name. I think I'll also use some spare capacity to run a Counterstrike: Source server for a while too.
....can it be a "HL 2 Deathmatch" server?
Counterstrike: Source and I don't get along. (this is, of course, a joke. :D)
So I got the new server parts last monday, put them together, and nearly went crazy trying to figure out why it crashed randomly all the time. Ram was bad, but worked okay when underclocked. Spent several hours each night in the last week configuring things. New ram came today, works great, so I returned the old stuff. Everything was going pretty well until this happened...
As an educational experience in 10 lines, I present the following:
How NOT to copy a system to a new partition:
mount /dev/raid/slash /mnt/newroot/
mount -o bind / /mnt/newroot/ <- this line should have been /mnt/oldroot/
cp -av /* /mnt/newroot/
***Notice that it's infinitely copying proc and device files
rm -R /mnt/newroot/*
***Notice it couldn't delete some files because they were "in use"
***Notice it says "command not found"
***Notice that none of your commands work because your entire system is now devoid of files
***Trace steps to figure out where you went wrong
***Bang head against desk
***Try to figure out way to salvage situtation
***Cry over week's worth of lost configuration work
***Fail at life
And I just now recall that earlier today I fried my CD burner (which was temporarily being used as the CD drive on the server) because I was stupidly connecting the power while the system was powered on. Not a big loss, as it was failing anyway, but it occurs to me that I can't exactly boot off a cd to reinstall everything without having a working cd drive. I've hot-swapped plugs like that hundreds of times, both at work and at home, and never had a problem. I've also never removed all files on my system by accident either. Both in the same day, and bad ram + my car's fuel pump dieing last week. What gods have I angred? Heh... at this rate, if I disappear in the next couple days, it'd probably be safe to assume I got hit by lightning or something. :-)
I'm blaming everything on Puxatawny Phil, who, with his one vicious lie, perpetuated a chain of events that have left so many people in a lurch. Mostly that it's still winter and bad things happen in winter. I've had two RL friends suffer technological problems too and a bit of me is getting paranoid. I hope your luck changes soon. You're doing stuff that to most people, myself included, is massively complicated. I don't think you fail at life. *digital cookies*
...wow... I thought I had been having bad luck..!!! (at least I've got my laptop and my home PC! =D)
I still don't have a date yet for the transition, since I'm still working on the new setup (software wise... the hardware's working great now) and don't know how long that'll take. As I said before though, I plan to give everyone lots of notice before the move, so they won't be surprised by it, and can prepare as needed. My plan for giving notice currently involves this:
1. I plan to email everyone at their xepher.net email address. If you're in Phase 2, there's no excuse, as you should either be checking that, or having it forwarded to a valid account. The email is likely going to include a link for you to click on that will verify your account as still active/in-use.
2. Accounts I don't get a response from... I'll look over the list, and ones I recognize or know something about, I will try to contact them myself or for people that are pretty active here, I may just whitelist directly. I'll probably look at web traffic and see if they're even being visited or not. Ones I don't whitelist will have their accounts locked with a notice page explaining the need to contact me.
3. Any accounts still not responding will not get fully migrated to the new server. I will (and already have) put a copy of all user data on the new server, but I won't actually add them into all the system files until they contact me. I'll probably keep that data around for at least six months, maybe more, just in case, but I'm probably gonna want a good excuse if you takes you a month or more to notice your site is missing.
So, to make this go smoother when the time comes, I ask all of you in Phase 2 to make sure your email is properly set up. Either forwarded to a valid address, or that you check it here at least weekly. Wouldn't hurt to send a test message to yourself @xepher.net to make sure it gets to you. I'll be sending an automated email from adminATxepherDOTnet for any of you who want to setup email rules/filters or such. That address is already whitelisted for the xepher.net spam filters. You may want to do the same if, like me, you use jihad-like levels of additional spam filtering.
I've got apache (the web server software) running on the new server. I've been doing a lot of testing and tweaking to see what's the best way to set things up. I tested php with the "regular" method we've been using here on the old server for a while now. The new server could run about 70 requests/second and that would full peg both cpu cores. I then tried fastcgi, and that was a huge improvement. 250/second, with only about 25% cpu usage. Finally I tried a new method/module where apache actually spawns a new process for each user. This also hit about 250/second, but only about 20% cpu. I tried a simpler test script, one that just echoed a single line. It still maxes at around 250/second, but cpu use was only 4% or so. The bottleneck isn't the CPU at this point, it's the network and the script itself. Still, the method change alone is about 4-5x faster, and uses about 90% less cpu time. On top of that, the new server (compared to the old one) is about 15 times faster. When I get this thing in place, we're going to have plenty of room to grow. Only limit is going to be bandwidth.
You may be wondering why the new method if it's not that much faster than fastcgi... Well, quite simply because EVERYTHING served from a user site can be served by a process running as that user. No need for fastcgi wrapper scripts, suexec and weird permission settings on php file. I can actually setup php as an apache module, pretty much building it into the server itself, yet it's still just as secure as what we've been doing. No hoops to jump through to make php files work. On top of that, you can secure regular files even more. Since the server itself runs as the user, you can set all your web files to be readable only by you, whereas before, they had to be readable by everyone in order for the server to get to them. It's simpler than before, faster than before, and more secure than before. That sort of thing comes along so rarely in this stuff that I just have to say it once again...
... and no, you don't have to have understood any of what I just said. I've just been nerding it up for the better part of the day, and I needed to rant my success. :-)
Ok, you say the only limit would be bandwidth? But does that mean its about the same as before?
Well, for the past week, bandwidth has been horribly lacking, as the datacenter's been having problems with their equipment. Normally speaking though, bandwidth is not the bottle neck. The server is routinely pegging out the cpu during peak hours, and that causes delays when loading pages... especially dynamic pages like forums and such. I believe the new server, combined with this new way of handling such dynamic pages, will be fast enough that the next bottleneck we run into will be bandwidth. That's a ways off though, and things would have to grow quite a lot for that to happen.
To give you an idea of what sort of bandwidth we have, I've clocked downloads to the server at over 7 megaBYTES per second. That's over 50mbps, compared to most home dsl/cable at around 1.5mbps, and dial-up at 0.056mbps. Of course, with this datacenter we don't have guaranteed bandwidth, but that's one reason it's so cheap. Still, our average user right now is about 3mbps over the course of a day, with peaks around 5mbps. That's at least 10x room for growth. I think the new server can comfortably handle us up until that point. If we get larger than that... well, frack, we're gonna need some serious income, and find a datacenter with gigabit hookups. :-)
...all this geek-speak is getting me all warm and fuzzy inside. ;)
Touch Fuzzy, Get Dizzy
Hey, a reference I actually get!
(The image won't load for me, for some reason)
...and now the image will load, too! Yay!
And I was right! I did get the reference!
I'm a little techno illiterate, but yah...
One thing though...My Xepher e-mail is CLOGGED with spam, so I don't have any redirecting to my personal e-mail Jenscripter(@)yahoo.com . Will all the information you'll be e-mailing will also be on this thread?
what happens to the old server btw O.o?
Hey psychobob you got coco on you frace :P
I will be posting MOST of the information here, as far as news and such goes. It's still going to be important to get the actual email though, as each one is going to contain a verification link unique to your account. It'll be a lot like the confirmation emails you get when you register on a new forum, and will help me automate the task of migrating all the account data to the new server.
As for spam, it looks like you've deleted most of it recently, since your inbox is really small. To keep it from piling up again though, you should turn on spam filtering. You need to set "Spam Control" to "On" for the filter level to have any effect. I notice you've got the level set to strict (1) but it's not actually turned on... that would definitely explain the spam you're getting. :-)
Well, I woke up with a not-so-happy email this morning. The SMART (hard disk monitoring system) daemon emailed me saying that one of the drives in the current server has failed a self test with a read error. It's the drive currently housing the root filesystem, not user data, but if it dies, the server is going down. Now, before anyone panics, these things happen with older drives, and things will keep getting worse, but it doesn't necessarily mean it's going to die tomorrow or anything. I feel pretty safe in assuming it will hold out until the new server is in place, but I wanted to give you all warning just in case I'm wrong. I'd suggest (as always) that you make sure you have current backups of your site/email/etc. just to be on the safe side. I've got a backup of everything I pulled to the new server about two weeks ago. I'll be updating that again in a few days again, as I continue getting the new server ready.
In the event things really go bad though, I'll obviously be unable to contact anyone though these forums or my normal email at xepher.net. Anyone that needs/wants to can contact me on AIM via "Xepher42" or at email me at gmail, where I'm also Xepher42. Here's hoping that's not needed though.
EDIT: I'm syncing all user data as we speak. I'll try and do this every couple of days or so until the new server is ready to go... just to be safe.
In other news, if you're having issues with stuff being super slow or failing to connect... the entire datacenter is apparently being DDoSed... basically a flood attack. It's causing packet loss of up to 50%, which really screws with connections. I hate hackers that do this sort of thing.
Will the new server have PHP safe mode on? For that matter, does the current one still have it? I've been thinking of installing Gallery 2.0 to revive my creaky, broken old gallery section, and it says in the requirements that it needs to have Safe Mode off.
We've dealt with this before, on my latest upgrade, but that was aaaaages ago :)
Ever since I switched PHP to cgi mode, it runs a script as the user who owns it, making it pretty secure (for the system, not necessarily the user) to run without safe mode. Basically, it means if there's some exploit that gets used, it can still damage any other files you have, but it can't hurt other users or the system. (In theory, of course...)
So yeah, safe mode is off... if it's not than I can turn it off. (I sometimes forget every setting I've made when I upgrade PHP and it replaces my config files.)
I.E. you're using suexe right? I'd hope so. :P
Perhaps just making some little backup utility for the user that tarballs their public html folder and then dumps their db and adds it to the tarball making it available for that user to download would be helpful. Something you and I can probably do with ease, but isn't quite as simple for other people around here.
Oh and why would gallery 2.0 need non-safemode?
Yeah, it's suexec right now, but it won't be on the new server. As I nerded out and talked about some other post (too lazy to look where right now) it's going to run peruser MPM in apache itself. Apache itself will fork a user-owned process to serve all content for that account... script or no script. This lets me use mod_php again, putting PHP directly into apache, instead of calling it as a separate runtime. It also eliminates the need for the executable bit and strict permissions as required currently by suexec.
As for the tarball thing... maybe on the new server, but for the most part, people should just be able to connect with their SCP or FTP client and then drag/drop their whole public_html (or even home) folder. I setup the filesystem with that in mind. Hence why each user has their mysql tables actually stored inside their home directory, although they can't write to those files directly, as that would be BAD, it does make for an easy backup scenario as I just described.
Gallery 2.2 has a huge thread about safe mode, but the main thing seems this...
QuoteYou cannot use pipes or redirects when executing system calls which is painful (though not insurmountable) for the graphics manipulation packages like NetPBM that expect to use pipelining.
mkdir() will not allow you to create new directories unless the parent directory has the same UID as the process being executed. This means that Gallery will not be able to create it's data directories under the top-level "albums" directory that you created using ssh/ftp -- your ISP will have to chown the top-level directory for you in order for you to successfully install Gallery.
You cannot call set_time_limit() in safe mode, which means that Gallery cannot do anything that takes longer than 30 seconds (or whatever the server is configured to allow). This means that at the 30 second mark, your operation will be interrupted. Since the most commonly used database (MySQL) in its most commonly implemented setup (MyISAM) does not allow transactions, this means that you virtually guarantee data corruption when your long operation fails.
The thread (found at http://gallery.menalto.com/node/3017 ) has a point about safe mode basically being what you use if you have no BETTER way to secure things... I totally agree, it's basically a workaround for a flawed model of webservers. That is, that the website and the webserver are owned by the same user, which is rarely the case in any multiuser environment these days. Suexec or the peruser MPM I talk about above are much better ways of doing security, and pretty much obsolete safe mode.
That said, I personally find gallery to be needlessly complicated code for what should be a rather simple task. I haven't used the 2.x series much, but about 40% of the bot/hack attacks I've had to deal with were from bugs in users various gallery (1.x) installs... another 50% were bugs in phpBB installs. I'm not saying don't use it if it's what you want. Just keep it up to date when they release bug fixes, and maybe take a look to see if there's some lighter weight code somewhere that still does what you need. I personally wrote my own simple gallery/thumbnail maker. It's one file, and does really nothing but accept uploads, let you title/date them, and automatically make a thumbnail. http://xepher.xepher.net/images/ Code is http://xepher.xepher.net/images/code.phps It's 4kb, and requires nothing outside of PHP itself. Gallery is 9.5 MB! and also requires external programs to process images. To me, that just means a lot of room for bugs and potential exploits, mostly in features and such that I'd never even use. I'm betting someone out there has written something that'll do what you need without being as much overkill as gallery is. Totally your choice though.
Quote from: XepherThat said, I personally find gallery to be needlessly complicated code for what should be a rather simple task. I haven't used the 2.x series much, but about 40% of the bot/hack attacks I've had to deal with were from bugs in users various gallery (1.x) installs... another 50% were bugs in phpBB installs. I'm not saying don't use it if it's what you want. Just keep it up to date when they release bug fixes, and maybe take a look to see if there's some lighter weight code somewhere that still does what you need. I personally wrote my own simple gallery/thumbnail maker. It's one file, and does really nothing but accept uploads, let you title/date them, and automatically make a thumbnail. http://xepher.xepher.net/images/ Code is http://xepher.xepher.net/images/code.phps It's 4kb, and requires nothing outside of PHP itself. Gallery is 9.5 MB! and also requires external programs to process images. To me, that just means a lot of room for bugs and potential exploits, mostly in features and such that I'd never even use. I'm betting someone out there has written something that'll do what you need without being as much overkill as gallery is. Totally your choice though.
I see your point, especially because my Gallery 1 install hasn't been a picnic for me to maintain either. I'm holding out hope that the 2.x series is less buggy and offers better separation of templates and code functionality (that aspect of series 1 was terrible
, even though it's both an obvious idea and easy to implement - when I asked for templates to be separated in atp autosite, the guy who made it redesigned and built it that way in a matter of hours). If it isn't, I'll be very cross.
I've thought of deleting the Gallery installation outright, but I can't make myself. There's something of value in having an image gallery on my site - something that isn't the webcomic or the blog - and I'm convinced that there's value in having it set up properly, with comments, an RSS feed and all the navigation and sorting you need when the image collection grows big. Your images script is a neat enough tool but looks a little too quick-and-dirty for what I want.
Gallery, on the other hand, is big and bloated, just as you say. Something inbetween should be good for me if I can find it - especially if I can import my existing Gallery data in there. Any tips, anyone?
Oooooooh nice choice.
I thought about that kinda setup for the small setup I've got separate from my Xepher.net account (it's my business server), but considering it's currently just a VPM rather than a full fledged server, I decided that it wasn't worth it.
Oh, I wasn't expecting you to be able to use my script... it's totally underpowered and hacktastic unless you use it EXACTLY the way I do. I just meant it as an example of how simple the actual image processing is, with built in php functions even. I originally meant that as a simple proof-of-concept while I added features like comments, navigation, tags, etc... But since it mostly did what I needed, I lost motivation and it stopped there. I'll probably pick it up again in the future, but, well... we know how that goes. :-)
Thanks Xepher. You're the best.
Not sure who you're replying to, I was referring to the server setup though. We used something like that in my apache admin course a few years back. It's not exactly super unique, and worked somewhat well for what we were doing with it. I'm not exactly sure how well it'll hold up under a larger amount of load though (although with some of the systems now days that's hardly an issue).
Oh, yeah, sorry... I was replying to Reinder last time.
Gotta love multi-topics in forums! :P
Hahahahaha!! OMG, all this time I was looking for that "on" switch in Squirrel mail and not on my account itself XDDD!!
I got dogpiled by so many e-mails, including....DR. PHIL!
Yah, I'm a dork :P
Psychobob: Yeah, that's often a problem. I've patched together a bunch of different packages -- forum here, webmail there, some custom scripts, etc. -- and people often assume they're all connected or related. I need to do something about that in the future, but I'm not sure what exactly. I don't really want to write my own webmail client from scratch.
well I finally got xepher mail rerouted to my yahoo one so hopefully its ok
had to sign up again to let ya know i am alive :)
my xepher mail is clogged with spam its impossible to get it to redirect at the moment
but my current email is working fine
Yeah Xeph, about your question on a mail server, I'm finding that Exim4 is friggin hot.
Really? I'll have to look into Exim again. I haven't gotten to mess with the new server for over a week now. Real life is hectic at the moment. I've spent both saturday and sunday working until 9pm this weekend. Oi! Good news is that the hard drive that failed its self test on the old server is now passing it again. Looks like it may have been a transient error.
Yeah the only issue I'm having is sending out to Hotmail. AOL, Yahoo, Gmail... all receive fine from my mail server... but NOT Hotmail. They just silently drop it. It never gets to the destination inbox, and your server never gets a delivery failure warning.
It has something to do with extra configs I have yet to look into or set up, but as a heads up it's detailed here (from when I contacted MSN tech support): http://postmaster.msn.com/Troubleshooting.aspx
It doesn't matter what mail server you set up, if it doesn't follow those guidelines your mail will not get into Hotmail, MSN, or Windows Live email accounts... and you'll never know cause you'll never get a response. :-(
Well, thing is, xepher.net does meet all of those guidelines. I have reverse DNS, SPF, etc. It's a real IP, and I can connect/send to hotmail. I've only had occasional problems with hotmail, lots of people here use them, and they usually get through. Gmail on the other hand auto-marks it as spam until the user puts a xepher.net email in their contacts. Weird.
I think I may have deleted my confirmation email along with the 3452 spam messages I received on my neglected Squirrelmail account. For some reason, the spam filter labels spam messages as such, but doesn't move them to the spam folder like I had set them to.
For the record, the account sabao is still somewhat active. Same goes for Yanilyn, since I manage her site.
Well, first off, no confirmation emails have been sent out yet, so you haven't missed it. Also, squirrelmail is somewhat, shall we say, "buggy." If you want automatic spam filtering, login to your account settings at https://xepher.net/index.php?page=account and use the email settings there to filter spam to another folder or just have it deleted. Please note, the settings there (and the filtering system they control) are COMPLETELY independent of squirrelmail. Those control/filter email for your whole account, whether or not you use squirrelmail, outlook express, thunderbird, etc. Squirrelmail functions simply as an email client (the same as outlook or thunderbird would) and thus, handles stuff after the mailsystem itself has processed/delivered it.
Looking at the system settings for your account, you have it set to filter spam (spam control = "on") and have it set to put spam in a folder (spam fate = "move to folder...") but you've left the "Spam Folder:" setting blank, so it defaults back to your inbox. If you put a folder name it there, it will dump the spam in that folder. Alternately, you can set "spam fate" to "delete" and it will just trash it all for you.
I hope that helps clarify things a bit.
is there an approximate time of arrival for the new server?
Well, I'm moving to a new apartment about the last week of may, and I promised everyone at least two weeks notice... And even though I think I'm just about ready, two weeks from now would put me too close to my move (where I may be without net access for a few days.) So I'm thinking it may be better to do after my move, to make sure I'll have internet running at my new place, as well as give people plenty of notice. Right now, I need to create an account confirmation script/page/email system to let people verify their accounts. I also need to figure a few more details of the email system. To be tentative, I hope to be sending confirmation emails at the end of this week or so, and aiming for the first week of june for the switch. I've got to confirm it with the datacenter, but I think I can get the move accomplished with near-zero downtime. It requires that the data center have room to keep both boxes online for a week or so while stuff transitions over. They're pretty crowded right now, so I've got to hear back from them first. Keep your fingers crossed. If it works out, there should be only a few hours of downtime (read-only mode) rather than days.
Sounds good :) I'm moving at the end of May too... but that's a whole 'nother story :P
That's awesome, Xepher! I'll keep an eye out for that email..I have gmail and sometimes, legit emails get shoved into the spam folder. :( I hope the datacenter lets you do that because that would be so much better.
I still can't fathom how you afford this!
Man... servers are expensive, and having a server, even your own, in any data center is pretty damn expensive. I'm not sure how you do it, but damn I'd love to know! (http://www.databits.cc/images/icon_lol.gif)
DMFA donations percentages ;) :P
Well, the new server is basically all the donations received over the past 3+ years, plus a small bit out of pocket. The majority of that is, like Fes said, from DMFA. As for the costs of hosting... well, I just chalk it up with the rest of my utility bills. If I can pay 60 bucks for a cell phone, and 50 bucks for broadband, what's another 110 for colo? When you actually budget for something like that, it's not too bad. Odds are you're never so close to border line that 110 bucks would make or break you. Either you've got a fair amount less, in which case you just can't even consider it, or you've got a fair amount more. So far, all it's really meant for me is that I still don't have the Digital Rebel XTi camera I've been drooling over for years. :-)
(Now if I could just sell this house...)
I still try to help with a monthly $5 on average ;) I think I've been fairly good about it. I figure if I would spend about that much on some other free site (with possibly less freedom/space), I might as well try to spend that here.
I need to get my Japanese bank account hooked up to my Paypal account. I have next to nothing in my home account. x.x *adds that to list of things to do this weekend*
Hey, no worries guys. While I certainly appreciate donations directly from users, the real idea is to get the viewers to start chipping in. They're the ones actually getting the free ride off everything. :-)
Also, when this house sells, I'll have more money... well, not than I know what to do with, but at least more than I've had at one time ever before. So don't worry about me too much.
Yeah, I know, but what Fez says makes sense, at least for me. I actually have a job right now, and a real one, not a part-time one, like all the ones before. I can, to some extent, afford to chip in. Who knows how long this will last? I'm looking at grad school in the future. Yikes. If I get accepted anywhere, I'll definitely be looking at a ramen and bologna diet with not much to spare on anything else.
Just a bit of an update for everyone here. I know things are taking way longer than I'd originally projected, and for that I apologize. Anyway, finally got to spend some good, solid hours working on configuring the mail and DNS servers. I'm switching to a completely new mailserver (courier) and it's taking me a while to even learn what all the options are, much less pick the ones I need, and get them all playing nicely. I have made some good progress though, and the mailserver bit seems to be functional. I can send and recieve email, login with IMAP, and pretty much everything else. There should be a host of new features going with this, including full SSL support on all email, and more comprehensive email options for people with their own domains. You'll now be able to have multiple email boxes (with separate logins/passwords) , mapped to any addresses (on your domain) you want.
On a nerdy-humorous side note, (which most of you won't understand, but that's okay) I spent about 2 hours trying to figure out why in the world the server couldn't talk to itself. That is, I can ping the server from my desktop, and I can ping the desktop from the server, but the server couldn't ping itself. Thought it had something to do with the virtual server context security and such... finally figured out that somewhere along the way, I'd managed to delete the loopback adapter from the default runlevel... meaning the internal network interface was not going. That I didn't realize it a lot sooner is probably one of the stupider things. I guess it's something so low-level, and so 'assumed' that it never occurred to me to check it.
Anyway, things are looking like they're gonna work out well, and once I get everything put together, it's gonna be a lot easier to maintain and organize than the old server. The next task is to integrate spam filtering into courier... Oi! After that, I think I'm just about finished. All that's really left is to rewrite some of the stuff like the account management interface to work with the new system. Then it's just a matter of transferring everyone's accounts and files. I've learned my lesson, and I'm not about to promise a timeframe yet, but I am actually getting somewhere. :-)
Oh, and like I mentioned before, I think I'll be able to switch servers nearly-live, with maybe only an hour or two of downtime. Keep your fingers crossed on that one.
*reads it* *fails* COOL! ^-^ The new email system sounds nice...I stopped using the mail a while ago because of all the spam.
Yeah, sadly a lot of people just started ignoring it, now the email system is choking, because it's trying to deliver mail to mailboxes that are stuffed with spam. Just opening some of those half gigabyte mailboxes takes a good chunk of system memory. I'm making spam filtering the default on the new system, with some safe values to start with. People can adjust the filtering level, and there's also going to be a rather nifty way to make it learn. Obvious spam will just get dumped in Spam, and deleted after a certain number of days. "Iffy" spam will be put in a "Probable Spam" folder. When people move a message INTO the spam folder, it will learn it as being spam. On the other hand, if they pull a message out Spam or Probable Spam into the inbox or any custom folder, it will learn it has ham (not spam.)
Perhaps if you set up your system to only use mailboxes for people who actually need them too on the new server?
Well, I TRIED to have my spam be redirected to the folder named "SPAMMY" but it didn't. So I just told it to delete right away and set my spam filter to 1
Yeah, I'm same as Fes, there. I tried, and it would just revert to a blank, so I just told it to delete it outright. Not like it'll block what you send, right?
Data, yeah, I realize now that doing "catch all" emails just resulted in people getting more spam. The new system will only set up additional emails if people choose them. Likewise, I'll have an option to disable email entirely.
Hmm... the forwarding really does work. Maybe I made the controls on that web page too confusing, as there's several that have to be set for the spam to actually get filtered to a folder. I'll make sure and double check things on the new system.
Dragyn: Yes, there's a whitelist for my admin address, it should bypass any spam filtering.
I kinda' figured. Best to make sure, though.
On the way to the new server, I've created new SSL/TLS certificates for the mail server(s) and web server. They are, as before, self-signed certificates, so your browser and email client are likely to warn you that there's no authority for the certs. Since they are merely verifying that xepher.net is, in fact, xepher.net, and don't apply elsewhere, it's safe to accept them permanently, and you won't be bothered by it again.
On that note, all mail services now support TLS (it's like SSL, but inside regular channels) so you can set your mail client to use TLS for enhanced security on imap, pop3, AND smtp (outgoing) connections. That way, all mail transactions (including passwords/logins) will be fully encrypted. All this is in place already, since the old certificates expired last week, I had to update the current server anyway.
So, a question for you all. Would anyone be upset if pop3 email access went away? All the "brilliant" spam filtering stuff I'm doing relies on being able to sort and scan mail in various folders (aka "Junk" and "Not Junk" like things) but pop3 provides no way to download/check/teach anything in those alternate folders. I just grepped though the last week of mail log files, and only three users have logged into pop3 in that time. My guess is it wouldn't be a big deal, but I figured I'd bring it up here for discussion before I commit to anything.
As for a status update... spent most of the day tinkering with the mail system again. I've pretty much got it finished, with all sorts of nifty config options able to be implemented, and that server-side spam "learning" thing I mentioned before. A lot more features I'm glad to be getting into place as well, but I won't nerd-out on ya'll again. Suffice to say, I've just about got the framework in place, so I know what user settings I'll need to collect via the account confirmation process. I hope to have that up fairly soon, maybe the monday, but at least next week. I'll be moving the week after that, so it'll be time where I can't be online much, but people can have the time they need to get their notices and confirm their accounts.
I wouldn't be upset...if it cuts down spam, it's worth it, IMHO.
Wouldn't bother me a bit. Go for it.
I'm only using the web client anymore, so POP3 could go. (even though I have much better luck configuring POP3 vs IMAP, I gave up trying to make my Evolution client at home handle both.)
PS: whatever you did to my spam filters worked!! Many thanks ^.^
Glad to here it. Even just a few years ago when I put this server in place, spam was more of an annoyance than the plague it is today. As such, I took the design idea of spam filtering being an optional luxury, rather than a necessity. Now half the email accounts here are unusable because the "luxury" wasn't turned on or enabled by a lot of people. Now that I've had plenty of time to use and evaluate the filting programs, I know how reliable they are (and aren't) so I don't feel bad making spam filtering on the new server non-optional, just adjustable and trainable. Hopefully everyone can actually use their email when they need it after the move.
I can write a really easy tutorial on how to use Mozilla Thunderbird for IMAP connections. It's actually pretty easy to do.
Data: If you get a mind to do that, it could be helpful, but hold off for now, as the new server's gonna function a bit differently.
Where I don't really see how imap would function any differently on the client side of things despite how the server is set up, no problem.
Pop3 merely allows users to download messages from their inbox to their local machine. All further processing/sorting/folders/etc are handled on the client machine. IMAP keeps most things server side, especially folders. What that means, is that I can create spam filters that are safer, because they can put "iffy" messages in a folder, rather than just deleting them. On top of that, the IMAP server has a nifty hack in it that allows it to run a program when messages are moved/copied into certain folders. What that allows, is for spamassassin's learning programs to be run on messages users decided are (or are not) spam. That is, if you drag/drop a bunch of spam from the Inbox to the "Junk" box, it will learn from those messages and be able to catch more spam in the future. If the same thing was done with pop3, there server wouldn't be involved at all, since the user would just be moving messages around between folders on their own computer....
And wow, I'm an idiot. I just realized that's not at ALL what data meant. I'm not deleting it though, as it's a good explanation for everyone anyway. To answer what I believe DATA's question was (related to doing a tutorial): The new server is using maildirs instead of mbox files. This particular server (courier) handles folders differently than the current server. The current setup requires folders that are totally independant, and you can't make subfolders (not if they have mail in them.) The new server allows for mail and folders both inside other folders, but also has a different namespace layout, using INBOX as the default namespace root... meaning if you don't setup your client right, all folders LOOK like they're subfolders of the Inbox. Under advanced server settings in the thunderbird config, there's an option for "IMAP Server Directory" which, if set properly to "INBOX" makes folders behave as before (and how I think most people would expect.) Since I'm going to be trying really hard to make things understandable to everyone, I don't want to have some people setup with their folders different, and later getting confused when I refer to a certain folder in the spam filter tutorials. (Yes, I know, I'm probably being pedantic.) Also, courier allows a default set of folders already subscribed to (on the server end) and I need to do a few tests to see how different clients handle that. Squirrelmail in particular borks on it, because it doesn't take the list from the server, but from it's own config... it doesn't see a trash folder in it's config, tries to make one, the server says "no, it's already there" and then squirrelmail just gives up.
Anyway, I just think there's enough potential for quirks to pop up, that if you're going to put the effort in, it might be less wasteful if you wait for the actual setup. Also, it gives you a chance to let me know of any bugs you come across in the new system. :-)
Just a quick update. I'm still moving along with things, but real life is hectic at the moment. I was supposed to be moving into a new apartment yesterday, but hadn't been able to get in touch with the new landlord for more than a week. Finally she called me today, and I'm getting a different apartment, and I'm not getting it until next friday. As I was supposed to be out of this apartment next thursday, it makes for a complicated situation. Once all this is settled though, I should be able to focus on the server again. Sorry for the (seemingly eternal) delays. :-)
Uh oh...that sounds like a tricky negotiation between the two...I hope you find something that works for you. Maybe pre-loading a moving truck/trailer and then just zipping over Friday morning?
Just a quick status update for everyone. I'm in my new apartment and I love having space to myself. It's great! I've been running a live test of the new anti-spam system, using dspam. It's about 93% accurate, with zero false-positives so far. It's a 100% learning based system, meaning it's effectiveness should get a lot better as you train it. Spamassassin (the current/old system) catches a few percent more spam... but I had numerous false positives though my time using it, and it still let some spam though. It's also about 10x as resource intensive as dspam. Anyway, point is I think I'm going to switch to dspam for the new server, so that phase of testing is pretty much done. People are also going to have the option of using their email here or not. This time, it'll be totally disabled if not in use, before, there was only the option to forward it, which was fine until the advent of spam. You'd think I was born yesterday, the way I set up some of that stuff before. :-)
Most everything else on the new server is running as well as I would expect. All I've got left is to rewrite some of the account management scripts to modernize things. Right now, I'm working on the script that'll let everyone verify their accounts for transfer to the new server. It's times like this that remind me just how poor of a programmer I really am. I'm having to read the manual for every function it feels like. :-) Anyway, I hope to have this working in the next couple of days, at which point I'll email everyone and ask them to verify their accounts. Basically, you'll be giving a valid (offsite) email, and checking/setting all your account/site information. Also, you'll be required to (re)set your password, so it can be re-encrypted for the new system. Passwords are now going to be checked for cryptographic strength too. I know it's kind of a pain, but dictionary attacks comprise about 2/3rds of the hacking attempts on the server, and all it takes is one person with an easy to guess password, then I'm stuck cleaning up the mess.
Also, I have the intention of doing away with the phase 1 and phase 2 account distinctions. I believe there will be VERY few phase 1 members left after the move, and most of those will hopefully have no problem moving to phase 2. If there's a couple that have issue with the newsbox requirement... well, I'll deal with it on a case-by-case basis. Point is though, everyone will have full access to all features.
Lastly, the newsbox. As I've mentioned, I want to revamp/redo it. It needs to accomplish four functions. 1) Get people to notice that a given site is part of xepher.net 2) Promote other sites here 3) Attract donations 4) Allow emergency "broadcast" announcements. I'm going to do away with the regular/boring "news" items in the newsbox. It'll only be used if there's some critical, site-wide message that needs to go out to most everyone. As such, that last requirement doesn't really need to be fit into the newsbox design, as I can have it add a big red box or some such when needed.
Now, on that though, I'm willing to do just about anything that accomplishes those three goals. I want ideas from ya'll, about what you think would be good. Featured sites need images, not just text, I think that's a given. Beyond that (and what I mentioned already) I don't have any ideas yet.
QuotePasswords are now going to be checked for cryptographic strength too.
Will you also be requiring people to change their passwords every x-many months? I know you didn't mention this before, but I was just wondering if this was part of the cryptographic strength.
Also, are you doing away with the counter? I notice you didn't mention it in the newsbox portion.
I think that the donation button should be on the same line as the title and possibly larger. I think it might draw more attention at the top, rather than the bottom of the box. Or, alternatively, have the button be separate (and graphical) and let users position it. With all the content in the newbox as it is, the donate button is overshadowed.
A few thoughts on the network of sites:
-A series of links to a category page that would then list the site with the description
-A gallery-type list of sites
-What DeviantArt does - a category dropdown and then a list of all the stuff in that category.
Passwords: No, I won't be requiring them to be changed regularly. That's not much use in a geographically diverse population. It makes a lot more sense in local/corporate settings, where people are likely to "loan" their password to someone they shouldn't. I'm not worried about that, just totally unrelated/unknown hackers being able to brute force a password.
Counter: Probably get rid of it, though I'll still keep statistics with it behind the scenes.
Another thing I thought about was the donation system. Honestly, we've gotten jack-all from the newsbox, and having this complicated system in place to share donation income is rather cumbersome. I'm thinking it may be better to not accept donations on behalf of users anymore. The thing is, the few donations that do come in that way are usually quite small. Small enough that it seems downright stupid to be splitting 5 dollars, and letting paypal take a cut twice. What I think I may do (and I really want some opinions here) is just ask users to support xepher.net as they feel is worthwhile, rather than a mandated 25%. Users that want to take donations can take them directly, and I'll just rely on good will that they might pass on a fair share. Conversely, I'll still be asking for donations though the newsbox and the main site pages, so some people will hopefully support the network as a whole too. How does that sound?
That sounds good! I'm just wondering how large the images would be. Is this going to be a 300x300 type ad, one of those long vertical ones like on Fes' site, a standard width banner, etc.?
Quote from: Xepher on June 08, 2007, 11:53:45 AM
The downside (to a few people) will be that it won't really be configurable anymore. It's going to be a set size/color/shape/etc. so people will have to redesign their pages a bit.
Awww... I've got mine all styled up and colorized too. :)
Counter's no big loss, as they're pretty easy to write.
Looking forward to the new server!
On newsbox size, I'm thinking something around the size of the current newsbox, or the keenspot one. http://www.egscomics.com/ has a good example of the keenspot newsbox in the lower left. I might do a little wider and a little shorter, but I'm open to input on what would work best for people.
Ah, cool. In the past, I've usually stuck the newsbox in a 200px column, so I was hoping it would be vertical more than horizontal. :) When you do account confirmations, should we also have an image prepared? Or is this something you've already (or will) take care of?
The sites I maintain at my day job (painfully, slowly!) finally are requiring I learn scripting, so this is an opportune moment. Dealing with these (on an over-secured intranet) makes me appreciate the well-mannered webhost you are!
As far as the donation issue, as far as I know, nobody has ever clicked the 'donate' link on the old phase I site... Maybe it needs to have a more prominent position?
PS: I'm working over in killeen this week. Who painted all the grass in Texas green this summer? I've never seen it _not_ brown before. :)
Griever: I'll keep in mind what everyone comments about size preferences. As for images, the new newsbox system will probably be put in place a bit after the new server goes online. I figure I should focus on getting our capacity back up to par first. As such, there will be a large notice given for the newsbox when it gets done, and there will be an easy upload form under you account settings to upload/change your ad in the rotation. I'm debating allowing "animated" images. I probably won't allow actual animation/gifs, but may allow multiple images (3 or) so that will scroll or fade during the time your ad is displayed... let you put more information in, and (reasonable, non annoying movement) helps attract attention.
What's the current status of progression, if I may ask?
Yes... sorry, been meaning to update. Between the time sink that is harry potter, and this deal I'm working on with my father falling apart due to water-drilling regulations, I've been a bit sidetracked.
Anyway, status is that pretty much everything is installed, working, tested. That is, all the server software such as mail, web, etc. What's left is scripting the management system to work with all the new stuff. I can reuse a lot of the old code for things that haven't changed (such as user password changes) but other things, like email/spam settings, have to be rewritten entirely. It's not really that much code, but because it's code that's going to be messing with low level, very important things like accounts and passwords, I tend to be uber-cautious. Last thing I need is some bad form submission blanking the password table or something. :-)
Beyond that, there's a couple of optional things I may or may not do before bringing the new system online. One of those is a stand-in/temporary newsbox script, since the old one isn't going to be used much longer, and I'd have to rewrite most of it to work with the new system anyway. Another is a basic facelift for the site appearance. Perhaps not a complete one yet, but enough that I know what basic shape it's going to take, so when I rewrite the management system it fits the new layout and I don't have to redo it later. Lastly, I'm not sure how much to script the account migration. For example, transfering email folders to the new system... only about 20 people actually WANT their email transfered, and it's a rather complicated process with a lot of variables. I may script it, or I may just do that part by hand.
Now, as for a timeframe... I'm gonna be working on it a fair amount tomorrow and this weekend, and we'll see how much I get done. I'm going out of state for a week from the 15th of august through the 22nd or so. If I manage to get this done this weekend, and maybe ship it off by monday or tuesday, I think two weeks should be enough time for a transition before I leave. I don't want to push it much closer than that, because a new system is bound to have small problems that won't be found right away. If I don't finish it mostly by this weekend, I may push it until I return. I'd plan to have it all ready to go when I leave, and would ship it as soon as I got back. That would put the new system online that last week of august. I'd really love to let it be the former option here, (I want this thing done!) but I can't promise that.
Sounds goodly :)
Well, it's turning out I'm not nearly as speedy at coding/html/etc as I thought I was. I spend 3 hours just trying to get a handle on how CSS position attributes function in the real world. I still can't figure out how to make a footer with left and right "columns" without resorting to tables. Anyway, point is that I'm not getting the server off by tomorrow or anything, so we're now aiming for the last week of august.
I end up using a container div and setting the middle div (I'm assuming this is 3 column?) to have margin: auto for the left and right areas. ;D
EDIT: I should add...there's probably an easier/better way, but I do miss table layouts sometimes. Just easier with tables.
I've found a site that has css examples for just about every box type - I'll post it when I get to work - in about an hour...
sample css for boxes
Thanks... that's a useful reference, though sadly it doesn't list the particular problem I was working at. I was simply trying two boxes inside a footer box. I wanted the same thing you see on http://xepher.net for the footer, but with divs instead of a table. The problem is that the "float" boxes that are so common for that don't expand the footer itself down the page, so any background you have just makes a thin band, and then the text flows well past it. I think I got it worked out though, apparently the "position" attributes work differently depending on the absoute/relative setting of the PARENT as well as themselves. Weird.
Things in css are all inherited from their parent element, that's how it's always been. Though it's not supported correctly in most cases.
Technically, if you set a font for your <body> it should transcend to all the contained elements, but for some reason this doesn't always work as intended. The issue is, well... Internet Explorer. Which rarely handles things "to standards" and always ends up doing things the way they think it should be done. But if you write something for IE and don't test it in other browsers, then it's technically broken since you've got Internet Explorer (7 and 6 is still widely in use), Firefox (1.5 and 2.x), Safari (Mac and now Windows too), and Opera. Opera is especially important since most cell phones and things like the Wii use an Opera web engine. So having it display correctly on that is a must if you want people to be able to usably access your site from mobile devices and such.
*prods the main topic*
Heh... Okay, status. Got most of the web redesign done, so we're gonna have a new look. Got most of my back-end scripts written for the new system as well, so the actual management functions seem to be working. I've also created all the accounts for the users being migrated, all I'll have to do there is sync their data once the machine is live. What I've really got left is a final software/kernel update (super easy, just waiting 'till the last moment) and to write a web UI to work on top of the management functions... basically the front-end that you currently see as the "Account Management" interface. Shouldn't be too hard, as I do have most of the visual design solved. I hope to ship the server off in the next few days, but that depends on how much real-world work I've got to do. The project we're working on it about to start, so it's usually not much to do, interspersed with calls to "go and drop off this super-important permit right away!"
cool :D because I have a new application being prepped ;)
Heh, can't wait. Got an application just begging to be posted.
I imagine you're not the only one waiting... Also, new applications won't be immediately open after the new server goes online. I need to make sure it's stabilized first. It'll probably be an additional week before I open applications... though I could probably open them right away, I just wouldn't be able to get around to reviewing them really for that extra week. I do apologize for this long delay. I never meant for this to take so long, but... well, I have no real excuse. I'll try and update status here a bit more often though.
Currently, I've got half the web UI built... mostly the functions for receiving and parsing form input for various account settings. I have to make the actual input forms/html now, and then test everything to make sure there's no (at least obvious) way for it to screw up. Hopefully this weekend, if I get the time to work on it.
One possible snag I've hit though, the datacenter has changed their rate/pricing plans (again) and now the lower priced plan (that's most similar to the one the current server has been on) limits to a 2U server. The case for the new one I've built is 3U, as the previous pricing plan said "4U or smaller." Additionally, they charged me (without warning) $20 more this month than last month. I'm going to get in touch with them, and see what they'll work out for me on both this overcharge and on getting the new server in place. If they won't flex though, the cost for the new plan that this server would fit on is $169/month... previously, I've been paying $109. Probably not a show-stopper either way, but I'd really LIKE to get it resolved without paying $60 more than I am now for the same level of service.
Don't they grandfather? Kinda bad business to just raise your rates and charge people more without notice, is that even legal?
Don't know about illegal, but definitely bad business. Last year, they raised it $10, but they sent out a notice more than a month in advance. I'm hoping it was just a clerical error, since they switched to a new billing system recently, and my grandfathered-in pricing plan hasn't existed for years now, so someone probably thought they were just fixing a mistake.
This is probably as good a time as any to mention that I'll have some money waiting for you when my first paycheck comes in. It's been a rough year for me but things are finally looking up.
yes, once I get a few things figured out on my end, I'm planning on sending more too.
Before I say "cool" and "thanks" let me remind everyone that with the new server, the donation system is changing (going away.) That is, I'll still take donations directly for xepher.net, but will in no way require any percentage from people, nor handle donations for their sites (like the newsbox currently does.) Thus, everyone will be free to take donations (or not) for themselves on their site, keeping the full amount. You only need to donate to me/xepher.net if you wish to. The official date for that policy change will be the same as the changeover for the server, but far as I'm concerned, it's pretty much in effect now. So Reinder, Fes (and anyone else)... up to you if (and how much) you want to send on to me.
I plan on keeping a donate button to Xepher.net on my site, although my past history of nil speaks for itself. :( But who knows...maybe things will change. I keep having my fingers crossed.
Yea, I don't have a big user base to get donations, but I hope to make some wallpapers for all three comics soon, for $1 donations... those usually work, don't they?
A quick update for everyone... I got in touch with the sales department at the datacenter, asking about putting the new server in place and my "overcharge." They've raised rates on all customers/plans by $20, and they claim to have upgraded their backbone capacity significantly to justify this. There was an announcement on their forums about the price increase, but it was labeled (and email notice sent) as "Capacity upgrade!" and you had to read the full article to see the bit about the fees going up. Not the best business practice, but I can't fault them too much for it. As such, the minimum I'm gonna be paying is now $129/month. They say I'm on a bandwidth plan for a 5mbps connection currently. The new plan they have for 5mbit is $119, but it limits to a 2U server. I talked with sales and they said they'd let me keep the current plan even with the new 3U server. The other option is moving up to the $169/month plan, which is a full 100mbps port. That's technically 20x faster, but as it's shared bandwidth, it probably averages only 6x faster, but the burst speeds should be closer to 20x.
They've also given me the okay for having a week of overlap, letting me do the nearly-live migration (probably only a few hours of downtime) that I wanted... though they are a bit backed up, so if I want to do that (and I do) it can't be this next week. Unfortunately, I'm headed to India Oct. 2nd for about two weeks. I don't want to put a new server online, then leave the country just a few days later. I really think I need to be around/available in case any unexpected problems crop up. I know a lot of people are waiting to file new applications, and I apologize, but you're gonna have to wait a bit longer. I get back from India Oct. 15th. I should be able to ship the server on the 16th or 17th, bringing it online as soon as it gets to the datacenter, probably the week of the 21st. There is a _small_ chance I can get some free time this weekend, and manage to finish prepping the server. If so, I may send it off monday, leaving it with about a week online to flush out any bugs. I'm also going to see if I can find someone to hand root too while I'm gone. If I can pull off those two, it could be online before the end of the month. Keep in mind, that's a _small_ chance, but I'd prefer it to happen that way.
On the first issue, of bandwidth/pricing... I'm really tempted to go for the full 100mbps plan. Right now, the webserver is averaging 2-3mbps, with another 0.5 for email/misc. That's an average though, not a peak, and I think the peaks could be significantly higher (aka, faster) if the capacity was there. Also, with the new server, I'm hoping will go through another growth period, and the extra capacity might be a good idea. I just need to see if I can budget another $40/month (or $60, considering it just went up $20 anyway.) I was REALLY supposed to be "rolling in it" (money) by this point, and not having to worry about things like "budgets." Ah, but such is life. :-)
Time for another update...
I'm just about done. My checklist is down to one major item, and then the last minute prep for shipping the thing. A few minutes ago I just pulled a wire from a running hard drive and everything still worked (yay raid!) and even rebooted just fine while short one drive. It's rebuilding the data on the "failed" drive right now. I've also done a cold-boot test of my kernel upgrade/reboot procedures, and it's got tri-level fallback in case of a boot failure. The automated backup system is all fully implemented, tested, and functional. It will automatically back up the entire main system (including all user files) every night. It'll keep daily backups for the full previous week, and then weekly backups for 6 weeks prior to that, all entirely automated. Thus, if anyone loses file, accidentally deletes stuff, or even gets their account hacked, I should be able to easily restore everything to the nearest day, or, at worst (if you don't notice for a week or more) the nearest week.
What I left is the one thing I've been putting off because I like it the least, and that is finishing the programming for the web UI to manage user settings. I will hopefully finish that tomorrow though. After that, I just have to set the IPs and other network settings I get from the datacenter and then ship it off. There will be the work of actually migrating the files once it's in place, but that should only take a day at most, and the actual downtime for the site should only be a couple hours. My plan right now is to ship the system off on friday, as I'm going out of town for the weekend, and it should go online monday or tuesday when I get back, with the actual switch-over happening midway through next week, probably wednesday or thursday.
I know it's taken me WAY longer than I ever planned to do this, and I apologize to everyone who's been waiting for new accounts and such. Hopefully everything will go as smoothly as I think, and we can get back up to full speed around here... start taking new applications and all that. Thank you all for being so patient with this whole process, and keep an eye out for the final notices about the switchover.
Don't worry about it man. Everyone here, far as I know, had free hosting and can't really complain about the time it takes to put a new server in place since the costs are nearly, if not all, out of your pocket at the moment.
Well, there's been yet another delay. I emailed the datacenter asking for settings on thursday, but they didn't reply until the end of the day friday, after I'd left town. Then they sent me to the support department to get an IP assigned, and they just now got back to me. They don't assign IPs anymore, they want me to give them root on the box and they'll set it up once it arrives. On the one hand, I don't like that, on the other hand, anyone that has physical access and/or controls the network could get in anyway. Hopefully I can get it shipped off tomorrow, but Wednesday at the latest now.
you'd think if they needed root they would've already set up their user... hey hope it goes smoothly though =)
you should get one of the *cough* artists to make a UI if you don't like design... i also hate that part of the coding ...
Speaking of coding... I've been doing a lot of it. I've decided that, finally being productive, I'm gonna do it right to start with, before I send the machine off. So, yeah, another couple days, but because I WANT to take longer, not because I have to. :-)
On that note, I'll let you all have a sneak peak at what's taking me so long.
For now, I just made a bunch of banners doing text-to-image. Once the server is live, everyone will be able to upload their own banners. I'd recommend getting the sizes/format correct on your own, but the upload script will handle resizing images as well. Let me know if anything is broken or unfunctional with these. If you report a bug, gimme as many details as you can, any error messages, and what browser/OS/versions you're using.
Note: I am noticing a bit of a placement issue in IE7... the image is a few pixels off from the border, but not big. I'll correct that later.
Also, these probably won't work that well in older IE browsers as microsoft didn't support png files until IE7.
Fixed the IE7 issue. Apparently if you don't declare a big fancy and specific doctype, IE falls into "quirks mode" where it renders boxes the same wrong way that old IE copies did. That's right, the default method for Internet Explorer is not "use worldwide and well defined standards" but "do it wrong in exactly the same way since windows 98." MSIE CAN DIAF! :-D
The news box is looking pretty good. Will it be possible to change the background colour?
And yes, IE is... A real pain.
You know what'd be awesome is making an ajax version which dynamically pulls a random banner.
This does. Well, server-side dynamically anyway. The "users" array on the real deal is populated via PHP as the script is included (as are the config and shape vars.) It's populated in random order each time. Normally the the tag would look like "<script src=http://xepher.net/newsbox.js.php?username=testuser&shape=box>" This let's it be up to date, random, yet still cycle as people would expect with the forward/back buttons. If I called the ad randomly in java, I'd have to build a "history" array to have "back" work, and check against it with each new image so as to not duplicate banners. I figured it's a lot simpler to just populate it with "select username from Users ... order by rand()" This demo version is hard coded (I just dumped the output from the script and changed the file_server var) because the current server doesn't have the correct database on the backend.
Very nice! I think the 200x300 is an excellent size. I suppose my only critique would be the Xepher.net/arrows that pop down. The opening image and the demo ad image you have is so nice, but the Xepher.net/arrows look like a throwback to 1996....
Heheh... well, sometimes my thoughts/taste in design are a bit dated. I don't know what's hip with all the kiddies these days.
In all seriousness though, that's the sort of feedback I could use right now. I can agree that the dropdown text is a tad uglier than I'd like, so I was thinking of reworking that already, but I'm not sure what to do with the arrows. What is it you don't like? Is it the gradients, the color choices, or shape? Don't worry about offending me, just tell me what you think would look good, and bonus points if you can link me some examples. Changing those images is quite easy, and I sure don't want to look like a fugitive from ARPAnet. :-)
I think I'm on the verge of joining you there, Xepher. Nothing like a cute little 5 year old kid going, "You're THAT old?" to knock the youth right out of you.
The thing I noticed first was how pixelly the arrows are. I don't know of the best way to fix that, but in a community of artists, someone's got to know the trick. Also, I think the white/blue combo really diminishes the presence of arrows, especially over sites that don't have an image. I think a solid blue with a thicker (3px?) solid white outline would help - it would stand out on light backgrounds and the white outline would help define it on dark backgrounds.
However, on more thought...the design reminds me of the slideshow controls that pop down, like on Google Photos or Flickr. Rather than having the arrows overlay on the sides, how about in a small toolbar down at the bottom? Could also throw in links for hosting info and donations. Kind of like
[ << Hosting | Donate >> ]
popping up in the same height style banner as the Xepher.net above. Would also clearly make the arrows stand out, but the downside is that it would take up more space. On the other hand, with IE7 becoming more friendly, the background of the overlay banner could be semi-transparent....
EDIT: Unfortunately, this is the only example of an arrow I could find... the thicker outline helps diminish the pixelly. You can see it here (http://linuxbasics.org/blog/uploads/arrow_process.serendipityThumb.png).
The other thing that might help is make the arrow smaller, but make it look more like a button. Some smoother, like this (http://jonas.seph.ws/stuff/drf_fp.png).
Smaller could work, and I think I may go with thicker borders too. The dropdown seems to have a much thicker border than the arrows, even though I thought I made them the same. I also think the drop-in text (XEPHER.NET) does look a bit dated. To me the arrows look okay though, and I'm not sure what you mean by "pixely." First off, what browser/OS are you using? This thing tends to look different on different browsers, so it's possible you're seeing something different than me. That doesn't mean it's invalid, just that it's not intentional on my part. :-)
For comparison here's a screenshot of how it looks to me (complete with new site layout.) FYI, I'm running firefox2 on Linux. If it does NOT look like this to you, let me know, and let me know what you're using, then I can try and fix it up for your browser. If it DOES look like this, then maybe just explain what you mean in more detail. I definitely plan to improve this still.
NOTE: Please view the screenshot at full/normal zoom and resolution. If your browser shrinks it to start with, it will look REALLY ugly. It's 1280x960, so unless your desktop is larger than that, you should be having to scroll.
I'm on Mac OSX with Firefox 2.something or other. 126.96.36.199, whatever that means. :P
Pixelly...the diagonals seem very jagged. Like you can tell where each pixel adjusts. Thickening it up should reduce that, but I'm not an artist, especially a computer artist, so I can't say for sure.
Hmm... it should be fully antialiased images, so it shouldn't look any more "jagged" than any other image that size. I've redesigned the overlays completely though, so check out http://xepher.xepher.net/newsboxdemo/ again. You may have to clear your cache or force a refresh or two to make it show the new stuff. The arrows should be a single color now (instead of blue/white and red/white) with a grey/white thicker outline. The dropin text image should have rounded corners now as well. Let me know if this looks better.
Those definitely look much better, IMHO.
Glad to hear it. All I had to do was stop coding for a day or so, and let the artistic side of my brain get back up to speed, then it made sense how obviously ugly it was before.
I was able to invert the appearance of the overlays (so the logo and arrows were always up and disappeared when you mouse over) in both boxes with some random mouse movement. Turns out you can swap them back and forth if you mouse into the box during the transition. I was able to do this in both FF 188.8.131.52 and IE 6.
I like the looks, they're nice and smooth. Though it looks like IE 6 doesn't like the transparency on the arrows
Well, the transparency on the arrows is only the absolute edges where it's anti-aliased. You see how bad it looks while they're fading? Well, my only other option is to make it look that bad all the time, or to not fade the arrows at all. As for how you managed to invert the actions... yeah, I actually found that out myself a few hours ago. I've corrected it on the server, but I won't be updating the demo for now... which brings me to my next post. (Which is TOTALLY back on topic!)
The new server (Arclight) has shipped! Whooo!!! I dropped it off at UPS a few minutes ago, and it's scheduled to arrive on Thursday. That means it'll probably go online the next day, and I'll make the transition sometime over the weekend, probably late at night (late in the U.S. anyway.) While I'm going to take every precaution possible with all the data, I'd still advise you all to keep your own backup of your stuff, on the super-remote chance something goes wrong. Backup both your files, and your email (if you use IMAP) and remember that the new server is going to be SCP/SFTP only, no more FTP. It'll also be IMAP or webmail only, no more pop3.
If all goes well, 99% of you will never notice the actual switchover. I'm hoping to keep it down to a couple hours of downtime, but that all depends on how fast the machines can transfer data back and forth. The old server has to dump the database of every user, and then all files have to be transfered, where email and the databases will be reintegrated into the new system. Only 30 people or so have email to migrate, which I have to do by hand, and that's going to take me the longest. The plain ol' files should be pretty quick and scriptable. Like I said though, I'll probably do this in the early morning hours, like 3AM (0900 UTC.)
Looks great xepher. You are the Man :)
New server has arrived and is online. I'm waiting on IP allocations for the VMs before I can really do anything with it though, which they say takes up to 48 hours. At least it arrived in one piece though.