Automated Email Rules

[Xepher]

I've recently had to put in place some new rules about automated email. A lot of you have forums or other scripts that use the xepher.net server to send out things like new topic notifcations or registration confirmations. The problem is that a lot of these aren't well checked for sanity. That is, bots drive by, attempting forum spam, and that results in a lot of undeliverable mail being sent out by your forums. This in turn gets the server blacklisted as a spam source. Currently, roadrunner is refusing all email from xepher.net because of this, and yahoo.co.uk has basically got us on notice. This is pretty much due to the actions of only one or two users here. As such, new rules are going into effect immidiately regarding this.

1. The sendmail binary is no longer accessable to users. That's the local program that sends mail. A lot of web scripts use it on the backend to generate mail. There's no way to easily secure it user-by-user.
2. Thus, all outgoing email MUST be sent via the SMTP server, just like the mail server wasn't local to the script. (Most forums have settings for this if you need it.)
3. Email will be allowed ONLY to local/xepher.net addresses (and any aliases you have on domains hosted here) unless SMTP authentication is done.
4. You can only authenticate with the email system if you actually use email here (that option is set up via user-services) and you have to use your username and password. The reason for this is that means rejected/failed email will be returned to YOU instead of bounced around to me. It also gives other system admins a notice of who sent the email if/when they complain to me about abuse.
5. Keep an eye on your scripts, and do NOT let them be abused. If you set your username/password in your forum config, it will let it send email to anyone, using your creditentials. Do not do this unless you're absolutely sure it's neccessary, and you're willing to take responsibility if the script gets out of hand. Make sure your config also includes a valid return email address so you notice rejected messages right away. Pay attention to those rejection notices, and don't let the system keep sending email to invalid addresses.

As stated in the main xepher.net rules, email abuse is not allowed. Whether that abuse is directly by you, or by a system you've installed doesn't really matter. The new configuration I've just implemented means that email can't be abused by any script (or person) without it having your password. Thus, if you want to be safe, configure your scripts to send to your xepher.net email address (or one of your aliases on a domain you've got here) and do NOT give it your password. This means you can't do things like have a forum send out registration emails or stuff to other emails though. Most forums have a registration option that let's the admin directly approve/register new members that you could use. If you choose to give your password to a script though, you must be vigililant about what it's doing. Most of you know I'm not too hard to deal with, and I know sometimes things happen that you didn't intend. That said though, I reserve the right to terminate your email access if I feel you're not being responsible enough with things. Usually, I try to give everyone at least one direct warning before I do anything harsh, so make sure you don't ignore it. (That means you'd better have a valid contact email for me to send a warning to!)


Alright, rant over... exiting mail-nazi mode. :-P

[reinder]

So how do you make Movable Type use SMTP? It appears to be completely dependent on sendmail for password recovery.

[Xepher]

Well, it seems movable type does have an option to use SMTP (you have to install a small add-on) but it doesn't support authentication... there's a number of posts from other users at other ISPs with the same problem. Their hosts now require authentication to cut back on spam. There was one post which suggested a way to hack in support to the MT system, and another saying it was an upcoming feature for a future version. I figure MT can't be the only script that's still lacking support though, so I decided to go ahead and just do an end-run around the whole problem. I've installed a sendmail EMULATING script that actually connects to SMTP. It took a little bit of tweaking, but I think I finally got it working. If your script is looking for the sendmail program automatically (via default paths) then the script itself shouldn't need any changes. Some scripts (and I don't know how MT does it) only find sendmail automatically during install. If that's the case, you'll need to change the sendmail path in the config to "/usr/local/bin/sendmail" (it's probably /usr/bin/sendmail or /usr/lib/sendmail currently).

The second part, which needs to be done for anyone using this method, is to create a config file (with your email address, username, and password) so the sendmail emulator knows what to connect with. Create a (text) file in your home directory named ".esmtprc" (e.g. /home/USERNAME/.esmtprc ) Note that the filename itself begins with a dot, meaning it's normally hidden, so you may need to enable view hidden files. The file contents should be something like...

Code: [Select]

identity = USERNAME@xepher.net
        hostname = localhost:25
        username = "USERNAME"
        password = "PASSWORD"
Obviously replace the capitalized parts with your own info. As noted in the previous post, you'll have to have your xepher.net email enabled (can be changed in user-services) to have access to the email server. This won't work if you don't. Additionally, if you try using the fake sendmail without this config file, it'll simply fail with an error about missing configs. I still suggest that anyone using scripts that support SMTP authentication directly should use the direct method, and not this sendmail emulation.

IMPORTANT NOTE: The .esmtprc file is going to contain your password, in plain text. Make sure when you create that file, you disable permissions for anyone but yourself. That is, chmod 600. This SHOULD be the default for new file creation, but some programs override the server default with their own, so check to make sure. If you're using a graphical file manager (like WinSCP) go to the file properties, and uncheck all permissions except user read and write. If "group" or "other" has read access at all, then anyone else on the server could come read your password.

[griever]

I found a plugin for Wordpress that uses SMTP with the option of authentication, which I checked.  I'd like to use it to process a form on my contact page, rather than having my address out there in the open.  This is the only place I'd use it, as far as I can tell.  Do you foresee this being a problem?  Or would it be advisable for me to just stick with the alias I have set up?

[Xepher]

If I understand you correctly, the form is going to mail TO you (and only you, with a fixed address that users can't change.) If so, that's not a problem at all. My concern is the potential for an automated system to send spam/junk out to the rest of the world. In fact, if the email you plan to use is here at xepher.net, you don't even need authentication. If you're using your gmail (or other email) though, you would need authentication, but as I said, there shouldn't be any potential for abuse so long as users (and spam-bot drivebys) can't set/change the address the form sends to.

Does all that make sense? I woke up like 15 minutes ago and my head's not quite screwed on correctly yet. :-)

[griever]

Well, I read "If so, that's not a problem at all" and that made sense to me.  

I think I understand but the whole intricacies are confusing.  What I'd ideally like to do is use Google Apps and have them handle the whole mess but I broke my domain last night when I tried.  Typical me.  I still might give it one more try but this is my back up plan in case it doesn't work.  Thanks!

[Xepher]

So, combining my reading of your other post and this one, it sounds like you're trying to change your MX (mailserver) record over to be handled entirely by google. If you do that, it's fine, but it will break any email handling here at xepher.net, so I'd ask you to disable email on your account here to avoid confusion. Otherwise, the XN server will still think it's handling your mail, and any mail that SHOULD be going to google from your site (or anyone else at xepher.net, including me) will still be delivered to the account here. Likewise, as I mentioned in the other thread, it'll be up to you to manage the DNS entries for your domain (since you're setting your DNS to godaddy instead of here), and things may break if/when I make updates.

[griever]

Xepher email disabled!  I might transfer the domain...I've read a lot of stuff online in the past day and if I understand things correctly, I should be able to maintain Xepher nameservers but also be able to just make the MX record changes.  GoDaddy doesn't let you do that.

[Xepher]

Not quite... DNS servers work like this... Godaddy is the registrar who you bought your domain through. They can ALSO provide DNS service though. These could be too entirely separate companies in terms of how this functions. I'll call one godaddy-registrar and the other godadd-dns to help keep things clear. You set the "authoritative nameservers" at the godaddy-registrar to be either godaddy-dns OR xepher.net, you can't combine them. Then, on that DNS server (or on the one here) there are tons of DNS records (such as A records, MX recordds, CNAME... etc.) that can be setup. Since xepher.net is relatively simple, with only one public-use IP (except in the case of a switch to backup systems) what you can do is set the A and CNAME records at godaddy-dns to point to xepher.net's IP, yet point the MX record over to google. (Theoretically, I could change the MX record here on xepher.net's DNS to make it work too, but as I've got to automate dozens of sites, it's easier for you to do it if you need that level of complexity) It should work just fine, and end-users will never know the difference. The one caveat is if/when I make changes to the DNS entries for xepher.net (and the various other domains hosted here) I obviously won't be able to change yours, since you've run yours through godaddy instead. It really should only come up if/when there's a server move, or a switch to backups in the event of some disaster. Basically, if that happens, just get in touch with me and then I can get you the info you'd need to change it yourself.

Make sure you test the email from your form, just to make sure that mail sent from xepher.net gets routed to the right place. Let me know if it doesn't.

[Miluette]

This thread is confusing me lots.

Not being able to send out emails is already starting to frustrate the small amount of people who regged at my forum and oekaki. Is there a simple explanation how to get around that?

[Xepher]

If your forum/oekaki supports SMTP authentication, set it to use that. If it doesn't...

Create a (text) file in your home directory named ".esmtprc" (e.g. /home/USERNAME/.esmtprc ) Note that the filename itself begins with a dot, meaning it's normally hidden, so you may need to enable view hidden files. The file contents should be something like...

Code: [Select]

identity = USERNAME@xepher.net
        hostname = localhost:25
        username = "USERNAME"
        password = "PASSWORD"
Obviously replace the capitalized parts with your own info. As noted in the previous post, you'll have to have your xepher.net email enabled (can be changed in user-services) to have access to the email server. This won't work if you don't.

[Miluette]

Ahh, I see. I don't think I can do SMPT with the oekaki, so I put that file up.
I found where I could for the forum. I'll assume it needs all that same info?
Thanks. <3

[Xepher]

Putting that file up let's programs that normally use sendmail go through SMTP Authentication. The forum should support it directly though, so yeah, in the config there you set up pretty much the same info.

[Miluette]

Hmm...
Someone signed up yesterday and said they never got an email for validation. That worries me, wooooorries, 'cause I also know someone signed up and forgot their password.

[Xepher]

Well, first thing's first... are just SOME emails getting lost, or has no one managed to get any email from that signup? Secondly, are you talking about the oekaki or the forum? If it's the forum, well that's the same thing we're using here, so I know it works (when properly setup.) The oekaki might have it's own issues though. However, if anyone has gotten any email from the oekaki, then you know it's setup right (unless you've changed the config since.) Likewise for the forum if anyone's gotten mail from it at all. If those both work, that means it's configured properly, but email still can (and does) get lost all the time. Believe me, I deal with this on a regular basis, but it's especially bad with automated emails. All emails sent by a particular piece of software are nearly identical in format. So if some ISP has received a lot of spam from even one compromised install, then it's quite likely it'll identify your install's emails the same way. The oekaki could be particularly bad, especially if it's not making "properly" formatted emails... a lot of poorly written systems trigger spam filters quite easily these days.

My advice is to test both of them yourself... Just sign up for a new account using an offsite email (like gmail) which you know works, and see if you get the messages. If you don't, well I can work directly with you to figure out why and test/reconfigure things. If you DO get mail though, then it means it's being lost/blocked/filtered for other people somewhere past the xepher.net systems (e.g. the user's ISP or email service) and there's not much that can be done about it, save maybe changing some things in the format of the mail to look less spam-like. Usually if you've left anything at the default values (like for the welcome message) you can change those and it can help.