Do we have FastCGI? And the terrifying prospect of a Movable Type upgr

[reinder]

I've been looking into upgrading my Movable Type installation. If you were around two years ago, you'll know why this idea fills me with dread: the new installation didn't stay up long enough under the onslaught of comment spam for me to have time to figure out what I'd been doing wrong that made the spam filters not work. It brought the entire Xepher.net server down with it.

Still, we're two years further now; a lot of work has been done at Movable Type to improve spam filter performance and integrate the anti-spam material with the core of the software, so it should now come with good spam prevention out of the box. Also, it now works with FastCGI, which should reduce the overhead - and it was the overhead during the spam attack that made the server crash, not the spam itself.

What I'd get out of an upgrade would be:
1) A more generous license. Actually, I have that now, because I downloaded a new version of MT to my home machine and accepted the new license. Since then, I've been adding new users to the blog.
2) Comments. I do believe that not having comments enabled is bad for a weblog, and it's especially bad now that I want to get into some more serious, substantial debate about a number of issues, from the political to the artistic and back, again. Comments can only work with functional anti-spam measures
3) Hopefully, faster performance.
4) A number of nifty features that may or may not turn out to be useful. I'd like to be able to tag blog posts for easier findability of the many, many old posts, for example.

There are other blog platforms I can use, including the CMS used for the comics, but I want to stick with Movable Type because I know it well enough to help other users of the blog, and because when I don't know the answer, there's a community of people who do. This is very different from my comics CMS where if I don't know what to do, only one other person does.

So my big question is, do we have FastCGI enabled so I can get those alleged performance benefits? And the bigger question is, do you reckon it's safe to carry out the upgrade:)

[Xepher]

Well, I just added the fast cgi module (mod_fcgid) so you're welcome to give it a shot. It looks like it expects fastcgi scripts to end with .fcg, but I'm not sure, as I've never used this before. As for the overhead, it wasn't that much on normal uses of the script, only when the bots invaded did it get ridiculous. Also, I've got CPU use limits in place now, that should keep it from running away like before, so I feel pretty comfortable telling you to give it a try if you want. Let me know if I can help.

[reinder]

All right! I'm sort of torn between doing it as soon as possible, preferably before the new ROCR storyline debuts on the 18th, so the new bloggers get a lively publishing environment in which people can comment, and waiting until I'm absolutely clearheaded so I can go through the procedure correctly, and also remember to do things like back up my blog content and my templates. It'll probably be soon. At which times are you available in case of a panic?

[Xepher]

Generally speaking, I'm online in the evenings on weekdays... Converting to UTC, that would be 2300-ish, until about 0600-ish. Weekends are semi-random, sometimes I'm home/online most of the day, other weekends I'm out with friends and don't even get online but for a few minutes. If you have problems, try to email me, as I check that even when I don't have time to view forums.

[reinder]

I think I'll have a shot at it tonight. I'll re-read the installation instructions and back everything up so I can restore the whole thing fully if it goes wrong. Any crisis that doesn't involve the whole server going down, I should be able to deal with.

[reinder]

By the way, I think to enable fastcgi, you need to make an association in .htaccess. I don't seem to have any local .htaccess files yet. Should I make one, or would you prefer to alter the global .htaccess to include

Code Select Expand

fcgi': AddHandler fastcgi-script fcgiFor now, I'm proceeding without, because renaming my files and changing my file extentions accordingly is likely to be an easy task, but a tedious one.

[Xepher]

.htaccess files are meant to be local, directory specific config files. That is, there is no "global" .htaccess file, just the main configs. And a similar line is already in the config.

Code Select Expand

AddHandler fcgid-script .fcgWhich makes .fcg files get handled by fastcgi. If your file names are .fcgi (or something else) you could just do that with your own .htaccess file, but you should use the line similar to mine, as fastcgi is deprecated, and fcgid is what's replaced it.

[Xepher]

http://xepher.net/chat/ if you want. I see you're working on it right now.

[reinder]

And I was, and I joined Xepher in chat because....
... the upgrade went very smoothly, right unto the point where I wanted to rebuild the output files and post something. That led only to 500 errors and general misery. Xepher and I spent 2 hours scrambling madly to fix this problem.
Movable Type's resource consumption is insane. The sort of CPU and memory use you'd expect from a professional image processing program, not a Perl app that reads and writes to a database and outputs text.  It would hit the limits Xepher imposed on processes running on his server, and hit them fast and hard. Fastcgi was not the answer here, for two reasons: 1) though it keeps the number of MT processes limited to one, that one process would have been enough to hit the limits; and 2) it didn't work.

What did work? Switching from Berkely DB to SQLite reduced overhead a lot, and I think changing the Category templates so the output files didn't come out so big also helped. These needed to be changed anyway, because my category archives are too big to be useful.The resource consumption is still huge, but it doesn't cause the processes to die.

Right now, I have a fresh install of Movable Type that works about as well as the last one, but I still don't have comments working. They are activated, but the comment forms send users to the Xepher.net front page and comments don't show up on the site. I'll try to fix that, probably by switching most of the site back to Movable Type's default templates. Updating Movable Type doesn't reset the templates anymore, but this time, I actually rather wanted that - to start the design from scratch.

I hope this information helps people upgrading MT in the future. I took up a lot of Xepher's time with this upgrade, so I might as well make it beneficial to others. If you were just starting with installing blog software, I'd advise you not to start out with Movable Type, which has longstanding problems that the company making it doesn't seem to be able to fix.  

Like I said above, though, I'm used to it, and I want to be able to support the seven other people who post on the blog as authors. So I'm sticking with it for another few years.

[reinder]

Xepher, did you do something clever to the server so that files named mt-comments.cgi would always redirect to the Xepher.net front page?

[Xepher]

*hangs head* I am so sorry, I didn't realize that. Yes, apparently that's how I blocked it from overrunning the server last time. I totally forgot that's what I did, and I know that must've driven you nuts for a while trying to figure it out. Turning off the redirect now. Try it again.

[reinder]

Hey, no problem... I'd actually worked around it by renaming the file, so comments were working by the time you read my message.

(My advice: Turn the redirect back on. As an anti-spam measure, it's still somewhat effective, even if it only weeds out the very lazy/clueless abusers. As long as people who want to enable comments on MT know that this redirect is in place. Then again, I don't know how many other Xepher users still use Movable Type at all.)

[Xepher]

Nah, it's okay to leave off I believe. I don't think anyone else has an MT install, and if they do, normal resource limits should be good enough to keep the server running anyway.

[reinder]

Arright. I got the first attempted comment spams in, and they've all gone to the Junk folder, so no rebuilds were done. Good.
I'm also getting search box spam, and lots of it. These have never bothered me much except that I found the idea of people spamming search boxes puzzling. It's probably just a case of the bots spamming anything that has a form field, compounded by the fact that some bright sparks have in the past thought it would be a nifty idea to publish their search terms on their blogs.

If any of these cause noticeable performance hits, let me know and I'll look for ways to deal with it.

[Xepher]

Well, I got some time to play with fastcgi and PHP this weekend. I've now got the main site (including forums) running under fastcgi. Let me know if you think the forums feel faster this way. They seem quicker to me, but it's hard to be objective when you've refreshed the same page a few hundred times already. :-)

User sites are not using it yet though, as it takes a bit of trickiness to get it to work. I have to tweak each vhost config by hand, and then add a customized wrapper script to each user's public_html folder. When I setup the new server (whose parts should be here tomorrow) I will probably switch fastcgi to being the default for everyone, as I have to write new configs then anyway.