Annvix Development Blog

Instead of waiting until 3.0-CURRENT, I’ve spent the weekend working on a new glibc package to put into 2.1-CURRENT. The timeframe to get the benefits of a new glibc (i.e. free SSP support) would be too long to wait, and since Annvix (currently) has so few users, now is the time to get all the serious breakage done. =)

Of course, it’s been interesting. Last year I had done the work to get glibc 2.4 into 2.0 but discarded that work. Of course, since then, I lost the changes to the spec so I had to re-port a bunch of patches and rework the spec again. Then I was having issues building on x86_64, so then I updated to 2.4.90 (a pre-2.5 snapshot) that was in Mandriva 2007.1. More porting. Finally, I realized that the latest stable glibc is 2.5, which a few distros are already using (latest opensuse, rhel, etc.), so I’m now working on using glibc 2.5 (the same version used in RHEL5, which looks to be 2.5 with a bunch of fixes and stuff from 2.6).

Phew! Well, I’ve been working on glibc almost exclusively since Friday afternoon, and while I had 2.4.90 fully built (but untested) for x86 and x86_64, I’m starting my first build of 2.5 on x86. There are a small handful of patches (from ALT and Openwall) that need to be ported yet, but I noticed that in ALT’s Sisyphus (which I’m assuming is similar to Mandriva’s cooker), they’ve got a 2.5 there as well, so I might get some of the porting “free” from that package.

The major benefits to using a 2.5 glibc? Well, SSP support is the number one benefit I can think of. The second is this puts us on par with the newer distros; I figure if 2.5 is good enough for RHEL, it’s good enough for us (likewise, any fixes that hit RHEL can be incorporated into our packages, which is why I opted to use their source/patchset for a basis). Oh, but there’s no support for the 2.4 kernel anymore (which is ok since we’re defaulting to 2.6 now). But, with 2.0-RELEASE, if you still wanted to use a 2.4 kernel you could, but as of 2.1-RELEASE you can’t.

Of course, this may mean that 2.1-RELEASE may be branded 3.0-RELEASE anyways since this would be a fairly significant update.

2.1-CURRENT has been branched tonight, so work will begin on it soon.

In other news, it looks like ibiblio has some disk corruption going on, so the mirrors might be synchronizing bad data. Please keep this in mind if you’re seeing some bad packages or things don’t look right. I’ve emailed the admins there and am waiting to hear back.

Being a little bored tonight, I was actually contemplating how much effort it would be to rework Annvix to run on a BSD kernel than a Linux kernel. I see that the Gentoo folks are working on Gentoo/FreeBSD which is supposed to bring all of the goodies of Gentoo using a FreeBSD kernel. I find that very intriguing.

For Annvix it becomes even more interesting. The only purpose of Annvix is on servers. Linux strengths compared to BSD are hardware/desktop support. I wonder how much that carries over in terms of hardware used in servers… i.e. SATA drivers, network drivers, etc. Which are superior? I’ve often admired OpenBSD and FreeBSD, and I look at how many security advisories come out for *BSD (kernel-related) to the Linux distros (kernel-related). I should do some searching on MITRE’s site to see how many CVE names match up there.

Anyways, I’m sure it would be a lot of work. binutils would need to be patched, everything would need to be recompiled, you’d have to track FreeBSD’s kernel development (which would be ok except I’d probably have to match their toolchain versions, etc.). I suspect most stuff like rpm and so-on would work well enough with some work.

Would it be worth it? I don’t know. The longer it takes the Linux kernel folks to push a supported/stable 2.6 kernel, the more tempting something like this looks.

Of course, you could do it another way too. Make Annvix something that actually installs on top of FreeBSD. In other words, you’d install a bare-bones FreeBSD and instead of installing their ports, you’d install the “Annvix system” which would be binary, rpm-based packages. Then the core system updates (kernel, toolchain, etc.) would come from FreeBSD and Annvix is just like a set of precompiled ports on top of FreeBSD…

Things that make you go “hmmmmm”…

Well, it is entirely possible to do a remote upgrade of an Annvix server using the upgrade script. I just did it for the church I attend (they have a backup server running Annvix) over the lunch hour. The server was 1.2-RELEASE when they left for lunch and was 2.0-CURRENT when they came back from lunch. And I didn’t leave the house (having said that, the server is 10 minutes away so if something did go wrong, I could go there and fix it). I’ve also noticed that I need to make the upgrade script a bit more robust because apt will report an error if an FTP session times out or it can’t retrieve a file (which happened), so I had to cut a few pieces out of the upgrade script that had already been done in order to properly finish the upgrade. Wasn’t difficult to do, just a bit of a PITA, so I think I’ll keep looping and ask if the user wants to continue anyways if an error like that is encountered.

I also went through all of my websites and set them up in vmware to test. Each site passed with flying colors without me having to update a thing, so the apache+php upgrade should work fine (of course, this depends on the web apps involved too). Keeping php-mysql intact paid off (instead of dropping it in favour of just using php-mysqli); there should be no php code changes for the mysql-enabled apps. Also, the mysql upgrade went well; I did dump and restore the databases and ran mysqlcheck on them and everything came out ok.

Courier-imap was a little more interesting, but still didn’t take much time. The old version didn’t have the userdb explicitly noted, but it worked, so I had to add “authuserdb” to the “authmodulelist” line in /etc/courier/authdaemonrc, but beyond that it worked without a hitch. POP3 and POP3S worked fine (I’m assuming IMAP and IMAPS will work as well but I didn’t test them; the big thing with those is the authentication). I still plan to try dovecot, however. Other than that one change to authdaemonrc (and making sure to install courier-authlib-userdb), I didn’t have to change a thing in the configs. I did have to run makeuserdb to regenerate the userdb (was getting a gdbm error), but I think that’s because my webserver is x86_64 and the vmware setup was i586.

The only web issue I encountered was with a php ical-based calendering app. For some reason if you just go to it without any arguments it defaults to 1969 instead of the current year. Not really sure why, but can easily be worked around by linking to it with the current date (nothing showed up in the logs). Keep in mind that the httpd.conf from 2.0 and the same from 2.2 are vastly different. I recommend making a copy of your old httpd.conf and merging in the changes to the new one. Also, if you use mod_security, keep in mind that mod_security2 is a little bit different, so you might want to keep mod_security for a quick upgrade and then look at migrating stuff to mod_security2 (keywords and such are completely different). I suppose it wouldn’t be too difficult to keep both around for the life of 2.x, but mod_security2 is supposedly the better of the two so migrating to it sooner rather than later, especially if you use it, might not be a bad idea.

All in all, I think the webserver upgrade should go nicely. Need to tweak a few little things in the upgrade script, but overall it went extremely well on both the webserver upgrade simulation and the remote upgrade of the church backup server.

In light of this, I fully intend to upgrade the webserver by the end of the week and then unless there is something critical that breaks, will be releasing 2.0-RELEASE next week.

One machine left to upgrade to 2.0 (which is the primary web server). Every other machine I run is now running 2.0-CURRENT; the last of the “big upgrades” (well, second-last) was done this afternoon (my LDAP/DNS/DHCP server for the LAN at home). Upgrade went smooth for the most part… a few changes were made to the upgrade script as a result of this one. A few notes:

The httpd.conf file is sufficiently different that you will want to use the new file not the old one. The old one will cause you grief as a few modules are missing (biggest culprit seems to be mod_access.so); it was easier and faster to start from scratch and merge in the few minor changes.

OpenLDAP seems to be compatible with data from the old version; I did do my slapcat first but didn’t need to remove the old database and slapadd the data; my authentication via LDAP worked without a hitch post-upgrade. There is one caveat, however. The new openldap will absolutely not run on a 2.4 kernel, so you will need to reboot before slapd doesn’t give you errors. Took me a bit to find that out (couldn’t figure out why slapd was dying when I tried to start it).

Samba wasn’t starting either, but that’s since been fixed in svn. For the record, removing the “guest” option to the passdb keyword made all the difference (otherwise smbd will segfault when you start it). Default configs have been updated, but you may need to change local configs.

Beyond that, the upgrade went flawless. I even did it “remotely”; the entire upgrade was done via ssh, rebooted, and it brought the system up properly into 2.0. Now, I’m not sure if I recommend a remote upgrade like that (seems awfully risky to me), but it looks as though it’s doable (how many other Linux distributions can boast that particular feat?).

Of course, please please *PLEASE* be sure to backup before you attempt an upgrade. Things like dumping mysql databases, openldap, etc. should be a must for everyone. I haven’t tried upgrading a running mysql-using system yet, but I think there it would make more sense to do a mysqldump and then import it post-upgrade (openldap was ok because both versions were 2.3.x so compatibility shouldn’t be such a big issue, but please slapcat first anyways). MySQL is going from 4.1.x to 5.0.x so I’d really recommend doing a dump, removing the old database, then creating it all fresh with an import. Likewise for postgresql.

Otherwise, it looks like the upgrade script held up quite nicely. This also means that if I don’t encounter any real problems over the next week, 2.0-RELEASE will most likely come out the first week of February.

After having googlebot eat up over 40GB by spidering the subversion repository (which also resulted in amazingly high loads), I’ve moved the subversion repository to another system that won’t use the same bandwidth (and CPU) from the main server. So it’s back up now. At the same time I made a few changes to the repositories. I’ve decided that having two repositories (packages and unpackages) didn’t make sense, so I merged unpackages into packages (placing them in a /removed/releases/ tree). Also, for those who had write access to the repository, the URL has changed slightly; it’s now only available over SSL (I’m not sure if it was before). The URL itself will still work, although using svn.annvix.org is probably preferred.

I’m going to try and setup an RSS feed for new commits as well, which could be interesting. Other than that, the machine is a little less powerful than the server it used to be on, but has twice as much RAM so subversion should play quite nicely on it.