Topics

UPDATE UPDATE: I've finally deleted all the non-responsive sites. Yeah, longest two weeks EVER. :-)
UPDATE: Sites who haven't responded have been disabled. You've got two weeks before I delete things.

It's that time of the year again... time to see who has abandoned their sites and stopped updating. I just sent an email to the on-file addresses of about 2 dozen users who have not updated in over a year. In a week, I'll disable said accounts if I haven't heard from them, and eventually they'll be removed if I still don't hear anything. Do note that this email was sent to the address on file... so it's a good time to remind everyone to log into user services and make sure your email address is up to date. If you find it's not, please update it, and let me know if you're worried that you may be one of the accounts pending deletion.

Well, I feel I owe everyone an explanation for the day-long outage yesterday. Let's start from the top.

For the past year, xepher.net was hosted on a virtual machine at a company I used to work for, because I got a discount there, paying $63 for what should've been $250 normally. They finally decided I've been gone long enough, that the discount has to go. So I look for new hosting. I find a place (call it option A), and it's $160/month for ostensibly the exact same level of service. I put in an order, and don't hear anything for a couple days, despite emails to their support and such. This isn't very promising, so I look at other options. I find another place (option B), really cheap, $55/month, with a slightly less powerful setup, but more bandwidth and diskspace, so I figure, why not? They get me setup pretty much right away, and I spend a day or so moving stuff over there.

I switch all the services on, and it runs great for a few hours, some hiccups, but not a big deal. After a couple days, I figure, eh, this'll work for now, and my contract is up in a day or two anyway on the original service, so I have to move, shutting down the old host and closing that account. I go ahead and let the order for A stay in place though, since I'm curious to compare the two. It comes through about 48 hours ago, 6 full days after I actually ordered it, and by then I'm noticing more and more performance issues with B. The problem is that the CPU is powerful enough, but they have too many users on one machine, sharing the same disk, so anything needing a file gets delayed really bad. Since we have about 150 users here, all with websites, this shows up as a system load of over 100... e.g. there's 100 people waiting in line for a file. At this point, I get a letter from B, saying other customers are complaining because I'm using all the CPU. This isn't the case, as it's the disk that's lagging, and it's not my fault. Other customers may not have 100 people in line, but... well, picture it this way. I have 100 people in line with bucks, to collect their files as they trickle out of the spigot, so my "load" is 100-in-line. The other customers may not have 100 waiting, but they're bringing dumptrucks instead... big file transfers, but since only one person is waiting, their load is only "5" or some such. I tested this by shutting down all services for a bit in the middle of the night, and the load was still sky high, even when my server was doing nothing. Anyway, besides the point... what happens next is the fun part.

The of B emails me, with the complaints... but instead of giving me a chance to respond or work on it... he just shuts down my server. And in the email he tells me he's installed a script that will automatically shut it down again anytime load goes over 10.  Keep in mind, there's already a line of 20 dumptrucks (each from a different customer) in line, but if I so much as get 10 people with bucks at the end of the line, I'm shutdown again. I can reboot the thing through the control panel, but it obviously mucks with stuff to just reboot every few minutes. I write out a reply, explaining all this, but he apparently went to bed, because I don't get any response for 8 hours. In the meantime, I do the only thing I can. I shutdown the web service and try to migrate stuff as quickly as possible from host B to host A.

And then it gets more fun... host A is even slower on the disk response than the first one. As in, I can literally download it from the internet faster than it can save it to disk, so it's taking like 18 hours to move all this stuff. I just kept at it for 6 hours or so, but realized this place will probably be even worse, and for $160 there's got to be a better option. Finally I got with ThePlanet, and get a dedicated server. No sharing it with anyone... no VMs, no VPS, pure hardware all for me! And it's $125.

I order it, but one catch, it doesn't come with gentoo, the OS I use. So either I rewrite all my management scripts and all the custom programming, or I find a way to install gentoo myself. The thing has two disks, so it should work, but... well... OS install via the internet. Yay. It's slow going, but I make some headway, but finally, I reboot and it won't come back up. I give in, and tack on another $30/month for a KVM (virtual keyboard/monitor access) to let me fix things. That takes another couple hours for them to setup.

So now I have KVM, and I work through stuff for a good solid 30 minutes, then it refuses to boot because I screwed up a boot menu option. No biggy, reset it and pick another thing from the menu. FAIL! I can't reset it with the ctrl+alt+del, since it's frozen. So I use their panel to power it off and back on... but my KVM goes dead and stops responding as soon as I do this. Later, I find out the KVM is powered by USB, and takes like 60 seconds to come up. So, by the time it powers up, the computer has already booted, and once again gone with the broken default!! So I put in a ticket for them to go reboot it manually, and wait some more.

They're actually pretty good on this, get it done in like 20 minutes, and I finally get a working, bootable system. Then I start copying all data off of B, because that's the last place a complete copy of xepher.net ever actually made it to. (Well, I have a full backup of everything on my desktop, but uploading from a home connection would take 3 days or more.) The attempted copy to A has been going for the entire day, and it's still not done either! I nix that, and just focus on moving everything into the physical machine. It finally finished about two hours ago. I've been up since 5pm on tuesday, and it's now the wee hours of the morning on thursday. 31 hours awake, and 25 of them trying to sort all this out.

There are a thousand more little things that went wrong of course, but I'm too tired to rant anymore. Bottom line is, while it wasn't my fault, I do apologize. I never like to get caught with my pants down, so to speak, and all this hit when my options/infrastructure was weakest. Almost any other time I have a backup plan that's better than this, but having to move only 2 days after you just finished one... and with a script rebooting your only functional system anytime you so much as look at it funny... Oi!

Well, with all that said, I'm off to bed. If you run into any problems or snags, it's entirely possible I missed 1 (or 100) things in this, so do let me know. Also, any of you hosting here... feel free to tell your readers/visitors what happened, and send 'em here, anyone is welcome to ask questions if you want clarification or whatnot... it's not just for members here.

Somehow, I keep thinking all this must've been 24 hours early, because I would swear this much fail would have to be an April Fool's Day prank!

Anyone here play it? I got it a couple weeks ago, and finally found a game I'm enjoying again (even though the server reliability sucks). Problem is, I don't know anyone else who plays. If you do, and want to add me as a friend, my solider name is Xepher42 (surprise!)

UPDATE UPDATE: FRAK! Well, that didn't go so well, did it? Long story short, we moved 2 more times in the past 24 hours. I think we're settled down finally. I've been up 30 hours right now, and I'll make a new post later tonight or maybe tomorrow that'll be LONG and explain everything that happened.

UPDATE: Move is complete. So far, bumpy ride, and there've been some issues and brief outages. I'm tuning things to try and make it work better on the new system. Please let me know if/when you run into problems.

Fasten your seatbelts, we're about to move to a new server. Xepher.net has been hosted on a discounted plan I got through my old employer, and now that discount is going away. As such, I'm going to have to find a new home for everything. Thankfully things are all virtual these days, so it should be just a matter of finding a new host and transferring the data. I expect it to be mostly seamless, with a few interruptions in service here and there, maybe a couple hours downtime at most. There's always the chance things could go south though, so I'm giving as much notice as I can here. I just found out a few hours ago myself. Worst case though, I do have full, nightly backups of everything on my own PC here though, so no matter what happens, all the data is safe, and I will get it all back up and working sooner rather than later.

The good news about still being unemployed though, is that I have plenty of free time to work on this. :-) The move may happen as soon as tomorrow, so by the time some of you read this, it might already be past tense. In any case, keep an eye on this thread for updates, and if xepher.net is unreachable, feel free to ping me for status updates on AIM/Gtalk/YahooIM... my screenname is Xepher42.

So I realized that I'm a geezer in internet years... I mean, "All your base" is my standby, and that's ancient, but before that, there was the original meme: All I can find is a remix version, but if this make you feel nostalgic, welcome to the dark ages. :-)

http://xepher.xepher.net/downloads/Hampton%20the%20Hampster%20-%20The%20Official%20Hamster%20Dance%20Song%201.mp3

Yeah, I take a lot of vacations... one of the benefits of being unemployed! :-) This time it's to Crested Butte, which just got like 3 feet of snow this week. Should be quite awesome. Anyway, regular comments apply... some internet, but I plan to be skiing and drinking, not web surfing. I'm leaving tomorrow (Tuesday morning) and 'll be back sometime between the 1st and the 3rd of March. See ya'll then.

So, I've recently decided to try my hand at baking. I know it's not the manliest of pursuits, but I love EATING baked goods, and I also love the DIY ethic, so I figured I'd give it a shot. I had a few mediocre successes -- bread that was decent, but really kinda plain -- but yesterday I finished something that's positively addicting. I mean it... I've been having to literally force myself out of the kitchen to keep from just chomping the entire loaf in one go. It's a sweet, white bread, perfect with butter. It also happens to be really easy to make. As such, I'm going to outline it here, in case anyone else wants to try it.


Ingredients:

2 cups warm water
3/4 cup white sugar
1.5 tablespoons dry yeast
1.5 teaspoons salt
1/4 cup vegetable oil
6 cups bread flour

Notes: Sugar can be slightly less if you want, and salt could probably tend slightly more. Bread flour is not QUITE the same as "all purpose" flour. It has more gluten/protein in it. It tends to make better bread, and only costs a few cents more, but you can make do with whatever you have in a pinch, though you may need slightly more of it.

Process:

1. Put the water in a big mixing bowl (from your power mixer if you have one, if not, biggest bowl you have) and mix in the sugar. We need it to be about 110° F after that, so microwave it if it's not warm enough. If you don't have a thermometer, it needs to feel hot to the touch, but not actually "burning". Think "The hottest hot tub" you can stand.
2. Add the yeast to the sugar water, by sprinkling it evenly over the surface of the water. Go play smash bros. for 15 minutes or so.
3. The yeast should make a brown foam about an inch thick that cover the entire surface of the water. If it does, you're good to go. If not, start over, and buy new yeast.
4. Add the oil to the sugar water.
5. If possible, mix the salt into the first cup or two of flour. You don't want to add it directly to the yeast/water, or it'll kill it off. If you're feeling lucky, just mix it in a bit at a time as you add the flour.
6. Add the flour a cup at a time or so, and stir it in evenly. It'll tend to clump, so you just keep stirring.
7. Eventually, it's gonna get really thick and paste-like. It'll pull away from the sides of the bowl and seem slighly lumpy. If you've got 5 cups or more in, it may be time to use your hands (my spoon was bending at this point.)
8. Mix in the final flour with your hands, kneading the dough and using the main ball of it to scrape the dry flour from the sides of the bowl.
9. When you get down to only a few spoons of flour left, find a clean surface (dough sheet ideally, or really clean/dry countertop) and sprinkle the flour on it.
10. Knead the dough for at least five minutes. This means smash it flat/away from you with your palms, fold it over, rotate it a bit, and smash it again. Repeat that a lot. You can add slightly more flour dust to keep things from sticking.
11. After five minutes or so, and when the dough seems evenly smooth, stop. Clean/dry the bowl, then oil it. That means spray it with pam or something, or rub with vegetable oil. Ball up the dough into a smooth, round shape and set it in the bowl, then cover it with plastic wrap, or, at least, a wet towel. Put it someplace warm-ish.
12. Give it an hour. It should double in size or so. If you have a plastic bowl and your house is around 70°f, great. If it's colder, or metal, it may take longer.
13. Flour your surface again, and dump the dough out. Put some oil on a sharp knife and cut it in half. Knead each half just a bit more (to get rid of the "cut" edge) and flatten into a rectangle about the size of a 9x5 bread pan. Fold the ends over, and fold the rectangle in half. Pinch the seam together and roll it a bit. You're making a lump slightly shorter and narrower than the pan. Oil the pan, and set the loaf in the center of it, seam side down. Do that for both loaves.
14. Cover the loaves with more plastic or wet towel, and let them rise another 30-45 minutes. They should rise about an inch above the top of 9x5 bread pans.
15. At this point, it's baking time. Preheat to 350°f (or 325°f in a convection oven) and put 'em about 2/3 of the way down. Make sure there's several inches of clearance on top (raise/remove the top rack) as the bread really rises high.
16. Bake for 30-35 minutes. Keep an eye on the top color. It should be brown but not burned.
17. Pull the loaves out, and immediately turn/dump them out of the pans and let them cool in the open so they don't get soggy.

To eat, cut a nice slide, and slather it in real, salted butter. It's absolutely delicious. After the bread cools for an hour or so, stick it in a plastic bag (simple grocery sacks work great) and leave it at room temp. Later, you can slice it easy, add some fridge-temp slice, and just microwave for 30 seconds (and spread the butter) to get that same fresh-from-the-oven taste.

Next time, I'm gonna try using butter instead of oil, for both the recipe and for the pans, as well as brush on some butter when they come out of the oven. It would be nice to get a little bit of that baked in, so you don't need as much added later. If anyone tries this, lemme know, I'm curious how it works for other people.

Will sent me this earlier... I laughed, and had to share.

http://www.chrisrawlinson.com/2010/02/kulula-airlines-flying-101-livery/

I'm off, once again, to go have some fun. Going to ski in Breckenridge for the week. Should be back on Friday (the 5th) or so. I'll have some internet access, so I'll keep an eye on things, I just won't be reviewing applications or anything. Email is probably the best way to get a hold of me if something comes up though. See ya!

EDIT: Finally found it... been digging on this for two days. Turns out, I made a really dumb mistake a few weeks ago. See below for updates.

Something is causing a huge spike in bandwidth use this month. I'm about to go over the prepaid amounts bought for the server. I don't think it's any sort of normal use that's doing this, as I'm having a hard time tracking it down. It looks like it might be caused by incoming data, rather than stuff served out. As a temporary measure, I've suspended all OUTGOING traffic on port 80. This means your scripts won't be able to grab remote images and such to process them. Mostly this may break some RSS parsers and such. If you have something that broke, let me know. Most of you shouldn't notice a change though. If this doesn't shake out the problem though, I may have to try some other stuff while tracking it down, and it could be slightly disruptive while I poke around in things. I'll let you know if anything major is going to happen though.

I haven't really seen much of it, but I'm intrigued by the idea. That said though, I fear it might not exactly be the next gmail. It reminds me in many ways of the early Microsoft "Active Desktop" push, along with IE's "push" websites. Nothing caught on there until years later when RSS feeds finally took off. Wave seems like it might be useful, but only in the same way AIM is... that is to say, it only works if everyone is using it.

Anyway, thanks to a friend, it appears I now have some invites to hand out. So if you're raring to get your paws on the newest google noun/verb, lemme know, and I'll toss a few out here. If, on the other hand, all of this was gibberish to you, then don't panic... you'll likely never miss it. :-)

It's a short/easy move this time though. From San Antonio over to Houston. Shouldn't affect xepher.net at all, and really, I just felt like making an announcement. :-)

Some of you may have noticed that many emails have been getting bounced when you try to send from xepher.net to other services over the past few days. Turns out the datacenter where the server is got their entire range added to the Spamhaus PBL. This is a kind of blacklist, and it specifically says "these IPs are dynamic (they change ownership frequently), for home and other use, and shouldn't be sending email." I requested an unblock a couple days ago, and it went through, but as they work to resolve the issue as a whole, it looks like the server got whitewashed with the rest of them, and back onto the list. I've again gotten it removed, and hopefully it'll stick this time. If not, I'll continue to work on getting it resolved. Believe me, this is more annoying to me than anyone... I (obviously) use this for all my own mail. :-)

I just saw "Where the wild things are"... I thought it was pretty good. I love the cinematography, and the way even the most absurd stuff (hitting owls with rocks) could be taken seriously. Anyone else have opinions?

Update: Second outage (0700 UTC Nov. 3rd) is not my fault. :-P The datacenter lost power and all servers there went offline. It's now back up as of 20 minutes ago.

My apologies for the outage today. The webserver was offline about 12 hours (exactly during the period where I was asleep.) It looks like it tried to restart as per normal due to monthly log rotations, but something prevented it from coming back up correctly. Unfortunately, I don't really know why, as it's an error I've not seen before.

It's back up now, and I'll be keeping a close eye on it. I'll try some troubleshooting later in the night to see if I can narrow down the cause. Thanks to all of you who IMed me letting me know there was a problem... If I'd been anywhere but asleep, I would've gotten to it much quicker.

Okay, so let's post music videos. They can be posted for being a cool video AND/OR being cool music. Bad videos with good music still count.

To start:

http://vimeo.com/6284199

This kid apparently plays half a dozen instruments, recomposes songs, and sings in 3 languages. http://freddegredde.com/

I, on the other hand, was proud to play Row Row Your Boat on a telephone. :-/

I've finally managed to go and review all the outstanding applications. If you were waiting for a response, please check the forums. A few approvals were done, and a few further inquiries. Having no day job is AWESOME! Now I just have to figure out what to do with my life.

So, the last tech job I had was chronicled in this thread http://xepher.net/forum/index.php?topic=330.0 and it lasted only 8 weeks and ended with much hate. This current job was way better, and I kept it the full year I promised myself I would. I gave my notice at the start of last week (written on my "happy one year at the company" balloon actually.) So this week is my last at the company. It's actually the longest I've ever held a job and I think I may even miss some aspects of it, and definitely the people, so it feels kinda bitter sweet. The problem was just the job itself... fixing computer problems all day for other people... was never going to be satisfying enough.

Anyway, I'm now going to be unemployed and looking for what to do next in life. I've saved up about $25K over the past year, so I have enough of a cushion there to move pretty much wherever and get started doing whatever I want. My thoughts right now are to move someplace "awesome" and probably go back for my Master's in Creative Writing... stick around at school and eventually become an english professor maybe. The main question I have is WHERE to go for that. Part of it is a school with a decent program, but the majority of the question is where I actually want to move to. I'd love to have some cool weather and mountains again, but mostly I want to live someplace that's good to live. A town that's not TOO big, but with some culture... a music scene maybe, but at least some people my age (late 20s-30s) that aren't all drop-outs or whatnot.

Bottom line is I'm really excited to do something new... just gotta figure out what. Any and all suggestions are welcome. Where's a cool place to live (if you could)?

Update: I'm back!

This time, I'm going to Costa Rica for a couple of weeks: July 17th through August 1st. While I'm gone, Will-san will be keeping an eye on things, though that's really just for emergency stuff. He won't be approving applications or anything like that.