Topics

Upgrade time again. Nothing specific, but prompted by a major security breach in a core package (XZ) used by almost every Linux (and BSD) distro on earth today. It doesn't look like any direct exploit or vulnerability happened here on xepher.net, but it made me want to update and verify things generally.

We're still on PHP 8.1 but 8.2 may be coming sooner rather than later. Don't expect any issues there, but, one never expects the Spanish Inquisition. That said, and for now, this round of updates should be "done." If you have any remaining issues or problems, please let me know. And I apologize for the interruptions the past 10 hours or so.

PHP 7.4 is end-of-life, and so mysql 5.7. The real impetus for this change though is that openssl 1.1 itself (which both of these depend on) is going EOL in September as well. Most people hopefully won't be affected in any serious ways, but if you have PHP scripts on your site (such as wordpress) you should make sure you're up to date with the latest version.

I will likely perform this upgrade sometime next week, but depending on how busy I am, may postpone it slightly longer. I aim to have it done by the end of July in any case, so please check your site software for upgrades soon.

EDIT: Upgrade complete. Please upgrade your software if you have issues, and reach out to me if you can't resolve things.

Starting next week (November 1st) there will be an upgrade to some of the core system software on the server. This will obsolete old passwords (set before around 2008) which are stored using weaker md5 hashes. Support for those is going away in the cryptography library used by service pieces of software (including SFTP.) Everyone who hasn't changed their password in the past few years will need to do so, or potentially be locked out of some services until they do.

To change your password, go to https://xepher.net/user-services/ and login with your old/current password, then change it. Note that if you use MySQL for things like wordpress, this will change your database password as well, so you may need to update your wordpress (or other software) configs. For wordpress, that typically just means editing wp-config.php. This would also be a good time to make sure that file is secure (it does contain your password after all) so click the properties for that file and set the permissions so ONLY user (you) have read/write access (for you nerd types, "chmod 600 wp-config.php".) Remove the access from group and other/world so no one else can view it. You should do this for any other web software you may be using... any file where you have your own password stored.

If you have any questions or problems, just let me know!

So the VM host we were currently assigned to at Linode was shared with other servers that were major IO hogs, causing major slowdowns to service here. I've upgraded/downgraded servers, effectively causing us to "bounce" to a new VM host with much lower IO demands, and this should vastly improve performance. So far, I'm getting astoundingly better benchmarks and results. That said, it may be necessary to do this again in the next day or three to explore other profiles. It takes an hour or two each time, so if you find xepher.net down, I ask for your patience. If it's down more than three hours though, something may be wrong, and please feel free to contact me. Try my xepher42 gmail address or any other way you know to contact me.

PHP 5.6 is approaching its end of life, and is no longer receiving regular updates from upstream. That means it's time to move on to the next version, which is PHP 7 (6 was experimental and doesn't count.) I've updated the server to PHP 7.2, and if you're using older PHP software, you may notice errors appearing on your site. If you see messages about mysql (specifically mysql_connect being an "undefined function") then there's a temporary workaround you can add. In any PHP file you need to still use the old MySQL functions, you can add this line right at the start of the PHP section (just after the <?php tag ideally):

Code Select Expand

include_once("mysql-shim.php");

Ideally, if you didn't write the code, you should just update to a newer version that supports php 7 and not use this. But until this, this workaround might help. If you have other problems (or notice errors on any of the main xepher.net pages) please let me know. My own code is using this shim, as it's ANCIENT, and I don't have the patience to rewrite it quite yet.

:UPDATE:

Well, it's more than a year later, and a lot of accounts have fallen out of date again. I've disabled these. If your site has a notice that your Wordpress was out of date, email me and I'll unlock it, so that you can upgrade.

Note that the current version of Wordpress is now 5.6.2.


:UPDATE:
Just as everyone seems to have finally been updated, Wordpress goes and breaks their auto-update process.

https://wordpress.org/news/2018/02/wordpress-4-9-4-maintenance-release/

Everyone (in the world) needs to update manually and then make sure auto-updates are enabled again!


----
Over the New Year we had an account compromised (almost certainly through a MILDLY out of date WordPress install) and it was used to attack other servers, leading to a warning from my VM provider.

Going forward, I'm going to be requiring that all WordPress installs be maintained and kept up to date. To enforce this, I've written a scanner that will automatically find and check the version of any WP installs, and it will be emailing me any time someone is out of date. If you have WP on your site, please go update it NOW to at least version 4.9.1.

As WP is a very popular target, even an install a week out of date can be vulnerable, and so I will be disabling accounts if/when that happens. I've already disabled several, and sent emails of warning to many others. If your account was one that was disabled, email me, and I can release it and allow you to update things. Also, please make sure your offsite contact email is kept up to date so that I can contact you in cases like this. Too many times I have to resort to disabling accounts to get the owners attention, and I'd prefer to avoid that.

Sorry to have to be so strict about this, but with the potential to leverage this server to attack remote targets, the threat is serious enough that I have no choice. If you have any questions or thoughts though, feel free to ask below, or contact me directly if you prefer.

Thank you all for understanding!

So, spam hit again, and I'm moving to a new mail server. As such, it required a lot of work to reconfigure from the old system. I think I've got most things back to working, but if you have trouble with email here, please let me know. The most likely remaining problems are related to people with custom email domains hosted here.

If you're using your own email client, you need to use SMTP authentication with STARTTLS on port 587 (the "submission" port) to send mail now.

Also note, as email is what's being weird, please feel free to post here on the forums if you're having problems.

EDIT: Your ~/.esmtprc file should look something like this now:

identity = user@xepher.net
        hostname = mail.xepher.net:587
        username = "user"
        password = "mypass"
        starttls = enabled

Many services were offline yesterday, between about 8am and 3pm Pacific Daylight Time. Everything should be fixed now.

The cause was a filesystem glitch that caused one of the disks to behave like it was out of space, even when it should've had many gigabytes to spare. This prevented MySQL from running properly, which meant eventually DNS went down, and also caused problems for mail delivery (as new messages couldn't be created in home directories.)

It should all be fixed now, but if you encounter more problems, please let me know.

My apologies for the downtime.

Going camping up in Washington's Olympic National Park tomorrow. I'll be offline through Friday, so my apologies if there are any troubles or help needed during that time.

Move is complete... at least as far as I can tell. This was a completely new installation on a new, virtual server, with all data migrated by hand. I apologize for the two hours or so of interruptions overnight, but I think most things are working now. I'm sure I'll have overlooked something though, so please let me know if you run into any problems. Email was especially irksome, so it may be best if you post below, just in case it's still wonky.

With the new server, I noticed a lot of spam was being sent via automated wordpress and other things. Apparently at some point I upgraded something and forgot to lock the email back down as I had in the past. I've done that again now, so if you're having errors with your PHP scripts being unable to send mail, you may not have a properly configured .esmtprc file. See this post for details: http://xepher.net/forum/index.php?topic=626.msg6855#msg6855

Also, you will get warnings about the SSH server key having changed. This is expected as it's a new server. Likewise, the cert for SSL email (imap/smtp) and https has changed as well. This is self signed and will generate warnings. This is expected, but obviously don't trust my server with your credit card info and stuff. :-)

So it's been several years since the last server move. In that time, the current datacenter has been bought out twice. Originally, it was "The Planet" then "Softlayer" and now it belongs to IBM. In all that time, hosting prices have come down and bandwidth has gone up, so in the next few days or so, I'm planning to migrate to a new host. I think I can squeeze us into a Linode VPS that will save me $85/month. On paper it should be faster and better in all respects except disk space. I'm going to poke at it for a bit though, and see how it holds up.

When the migration finally happens, it should be "transparent" to end users. Site owners may find themselves locked out for a few hours while a final file sync is done, and MySQL will likely be put in read-only mode for the final few minutes of transfer. I'll try to do all this in off hours as much as possible of course.

I'll post another announcement when things are done, and then you can all yell at me about the things I screwed up or forgot. :-)

I somehow left the forum in maintenance mode for a while here. My apologies to anyone that was unable to access things. It should be back to normal functionality now though. Please let me know if you run into any problems though.

PHP, Apache, and MySQL (along with a few dozen supporting libraries and such) have been updated. All of these were minor version updates, so shouldn't impact much, but if you run into problems, as always, just let me know.

In the not too distant future -- that is, a few weeks from now -- I will probably be upgrading PHP from 5.4 to 5.5. As far as I know, this shouldn't be too big of an upgrade, but you might want to make sure your PHP scripts and applications are compatible with PHP 5.5.

So I'm now comfortably settled in Bend, Oregon. I've two different internet feeds (I'm a nerd) and more free time than I know what to do with. If anyone is journeying near this part of the country and would like to say hi, I've got a spare bed to offer. Otherwise, know that all things are on cruise control as far as the site is concerned. I'm spending my time getting in shape, doing some mountain biking, hiking, etc. and I'm toying with Aruduino and the other electronics, as well as nerding it up in some local RPG groups and Belegarth (which is hitting people with padding sticks and calling them swords.) :-)

So, I quit my job on friday, and my lease here in Houston is up on the 18th. After that, I'm packing everything I own in a trailer and driving northwest. My goal is Bend, Oregon. I have no job, no place to live, and know absolutely no one for hundreds of miles in any direction. My only plan is to "chill the buck out" for several months and try to find something more interesting to do with my life than sit behind a desk.

How will this affect the site? Well, it shouldn't. I've plenty of money saved up, and the server bills are on auto-pay. Theoretically, I shouldn't need another job for two to three years. Only possible impact is if something technically mucks up while I'm in transit, I might not catch it for a bit longer. I will have a laptop with me and random wifi along the way though.

That said, I don't plan to work for at least another six months, maybe longer. I'm going to travel, relax, and enjoy things. To that end, if anyone is situated between here (Houston) and Oregon, and would like to have a pint of beer or a cup of coffee, let me know. This road trip may be solo, but it should still be epic, and I'd love to meet any of ya'll I can along the way. :-P

I've upgraded both the forum and webmail software to new versions. The webmail one went smoothly, but I hit some bumps with the forums. I think I got it sorted out, but if you run into problems, please let me know.

Over the past two days, /var filled up, preventing new email messages from being accepted by the server here. I've corrected the problem, and as messages queued on other servers are retried, they should start trickling in here shortly. If you normally have a decent volume of mail to your account here, some of it may come in big chunks now that things are back online. I apologize for the interruption in service, as I should have caught that earlier but I've been busy with the holidays.

As to the root cause, it was the apache error log filling up very quickly with deprecated and strict code warnings. Even though these are disabled in the default config, several users appear to have code the explicitly turns some of this extra logging back on, and thus fills up the log faster than usual. If you have a site here, now would be a good time to see if it's generating errors and update any third-party scripts you have installed to ones compatible with php 5.4. Not a big deal at this point though, I'm just going to rotate the error log more frequently.

The server has been upgraded to PHP 5.4.6 (from 5.3.15) and other minor software bits were upgraded as well. As usual, if you run into problems, please let me know.

I'm going on a cruise this week, and won't have any form of internet access until I return on the 7th of October. As such, please don't be offended if I don't respond to email or other issues until then. :-)

Edit: Back now, all good!

I've once again upgraded the system software. PHP is now 5.3.13 and apache is at 2.2.22 now. There were a few dozen other packages updated as well, and some obsolete packages were removed (mostly perl modules.) As always, if you run into any problems,  just let me know.