Annvix Development Blog

Due to issues with compiling Tripwire for x86_64 (or even at all with more recent gcc versions), I’ve been playing with AIDE for two days now. Like Tripwire, AIDE is a filesystem integrity checker. Unlike Tripwire (or, at least, the GPL version of Tripwire), AIDE is in more or less active development, although the last full release was a few years ago. They’re doing development on it still, even though it’s slow.

For the most part, AIDE works well. Documentation is inconsistent, which is a problem. It doesn’t use any encryption to protect the database or configuration like Tripwire, which is a drawback, but can be somewhat worked around. All in all, after spending some time tweaking the config and also writing some helper scripts and patches to the AIDE code itself (0.11-rc2 is what I’m using), I think I could actually use this.

I solved the problem of a verifiable database by using gnupg in my helper scripts. Works fairly well; need to tweak it a bit more. The problem with AIDE is that the database is a flat-text file and someone obtaining root privs could overwrite/manipulate it their desire all they want. There is a contributed script to encrypt it with gpg, but the way the script is written you need to have gpg setup without a passphrase which is the worst kind of stupid. I mean, really, if someone can get root privs and manipulate your db anyways, encrypting it with a passphrase-less key gets you nothing. I can’t even call it a false sense of security.. if this was thought out, you’d realize it was no security at all. And with a passphrase-protected gpg key, you can’t run an “aide –check” from a cronjob or anything because you’ll need to decrypt the file before you can compare the current state of the filesystem to it.Instead of encrypting the file, I opted to have a detached gpg signature associated with it. This isn’t perfect, an intruder can still read the file. But that won’t give them much. With the helper scripts, the signature is checked before a check or before an update of the db to make sure it hasn’t been tampered with. Of course, this pretty much means you need to use the wrappers to do anything. And you don’t need to have a passphrase-less key to use it. You can verify the sig without providing a passphrase; you just need to provide one when you update the database. Which is good… it means no one can update the database without knowing the passphrase.

The big problem with this is that you can circumvent all of this by not using the wrapper scripts. This, of course, is a good reason to have the aide binary stored on a floppy or CD-ROM (along with the database). But that causes it’s own usability issues. The real saving grace here would be to have some MAC definitions involved, via RSBAC, SELinux, or whatever. Not quite ready for that yet, tho. So will have to make do with what I can… at least it will keep everyone who isn’t root out of it. It’s not perfect, and not nearly as good as Tripwire in that respect, but until AIDE gets the cryptopgraphic checks/support built-in, this is probably as close as you’re going to get (once RSBAC is involved, it would be a lot easier to protect).

That just goes to show that an integrity checker like Tripwire, or AIDE, is a good tool to have, but isn’t the be-all and end-all of a host IDS. Just another piece of the puzzle.

Anyways, hopefully they consider my scripts and patches worthwhile to add to CVS (the patches don’t do much other than pretty stuff up.. the default reporting for AIDE is kinda ugly and cramped and don’t get me started on the –help output).

Well, piss me off, but after about 3 weeks of rebuilding packages (with other changes, mind you) SSP still isn’t 100% and I’d probably have better success going the Gentoo/OpenBSD route of putting the __guard and __stack* symbols in glibc, but the last time I did that and started having real problems I had to bootstrap my build off a fresh Mandrake 10.0 install. I managed to get about 99% of Annvix built with SSP, but the kernel refuses to build, as does syslinux, and I had to do some serious massaging to make grub and memtest86 build.

I like the idea of SSP, and I think it’s fantastic (and others do too, obviously, as it’s going to be a feature in gcc 4.1 mainline), but the maintenance of trying to make it work for me now, is too much. So I embark on another rebuild to get rid of it and then I think instead of wasting time on trying to get SSP working with gcc 3.x, I’m going to fiddle with RSBAC instead. Or a php-based configuration/monitoring solution so I can get a peek at my servers without using ssh (useful for when I’m on the road and have left my usb keychain drive thingy at home and can’t ssh in).

SSP is, I think, a lost cause for me at this point. If I knew more about gcc and glibc I could probably make it work, but it’s too much work for someone who basically knows squat about gcc and glibc internals. C’est la vie (hey, if I spelled that right, it means I’ve learned some french in the last 5.5 years!)

I feel I need to clarify my purpose behind the development of Annvix. There was recently a rather… heated… discussion on an internal (well, semi-internal…) Mandriva mailing list regarding the oh-so-useful %_mkrel macro. I won’t go into details on it here because, frankly, there’s no point, but as a result of some personal digging (hi Buchan), I’ve decided to make abundantly clear my purpose for Annvix so that some people don’t misinterpret what I’m doing with it.

I started Annvix back in late 2003 (November sometime, I think) and based it on Mandrake Linux 9.2 because I wanted to use the gcc SSP patches to compile everything (for those who don’t know, this offers stack protection to everything that gcc compiles with the -fstack-protector(-all) flags). To me, this was a very cool thing. But to get this into mainline Mandrake Linux would have been a (futile) fight. So I did what any competent back-scratcher did… I forked it and started my own thing.

However, from day one, my purpose has not been to supplant Mandrake (now Mandriva) as a server distribution. It has not been to create a new distro with thousands of users. It wasn’t even to provide any form of compensation or income for myself… for the most part, I love my job with Mandriva… it’s been a, for the most part, enjoyable 5.5 years. The idea of Annvix (at that point, OpenSLS) was not to provide or promote any kind of competition.

The bottom line with my fork was to provide something for myself. Previous to starting OpenSLS, I was running and maintaining rpmhelp, which was a site where I distributed a number of rpm packages for Mandrake; things like qmail, djbdns, exim, etc… things that couldn’t or wouldn’t be in Mandrake (think PLF but with a different goal). rpmhelp was cool, but maintaining packages for various different platforms (I think I did it for Mandrake 7.2 through to 9.0) was tiresome. If it was just for me, I would have done only for those platforms I actually needed the packages on, but many people enjoyed the packages so I carried on. After 9.0, however, I just didn’t have the effort to maintain it anymore.

But when 9.2 rolled around, I was still running Corporate Server 2.1 on my public servers and, to put it bluntly, at that point it was aging. I wanted apache2, newer PHP, etc. I ended up rolling my own packages on the side for my servers. And then, again, people wanted to partake of the fruit of my labour. No biggie… rpmhelp was there, so I used it.

It should be no surprise to anyone that kernel updates at that point were… less than forthcoming. There were numerous problems with getting updated kernels out in a timely fashion and being the security idiot that I am, I found myself disliking the thought of continued Corporate Server 2.1 use because of the age of the software, and I didn’t want to move to Corporate 3.0 because I didn’t have confidence in the speed of kernel updates (over which I had no control). As an aside, kernels are much better now… I would feel more than confident running Corporate 3.0 on a public webserver now (the same couldn’t be said 2 years ago).

So I decided that not only did I want to have kernels that were more up-to-date, more up-to-date software, but I also wanted less bloat. Let’s face it folks… any non-specialized Linux distro is loaded to the tits full of useless bloat. If I want a server OS, I don’t want X. And if I don’t want X, I shouldn’t have to have the bloat that some programs compiled against X bring in. So my goal was then to create something current, and something without a lot of bloat. And that’s where OpenSLS came from.

So if this was just for me, why did I make it public? Well, for one, it was a proof-of-concept that a distro could be fairly easy to use (like Mandrake), but slim, trim, and secure. It wouldn’t be much proof if only I could see it. So I made it public so anyone could poke through CVS, use the rpms, whatever. It was meant solely for myself, but it was public in the spirit of open source.

As time went on, OpenSLS turned to Annvix, and Annvix drifted further and further from Mandrake in both technical merits and goals. Yes, I still track cooker packages and pull in patches and changes and whatnot from Mandrake. After all, I work for Mandrake, so it made sense to base stuff on it… and I like Mandrake. It didn’t make sense for me to start observing Fedora or Debian and basing stuff off of that. I spend too much time in the world of Mandrake (I guess I should start using Mandriva now, although it still comes akwardly to me) to muck around with something else.

Oh, I should also note that this is all in my spare time and for 98% of Annvix’s life, I’ve been the sole developer. So momentum is understandably slow (sometimes slower than I’d like, but heck, I need to eat, right?). Work comes first, be it Mandriva, web development projects, other consulting projects, whatever. So there have been some extremely slow periods with Annvix development as a result. Not to mention that tracking cooker on every change would be a big job in and of itself.

At any rate, Annvix is my play-thing. Yup, you bet I use it live, and you bet it works quite well. I’ve had people ask why I don’t contribute this stuff to mainline cooker. Well, that’s an easy one. 50% of my ideas and proposals are either debated to death or fall on deaf ears. Makes encouraging development rather hard. The other is that a lot of my ideas don’t fit with the Mandriva philosophy. For instance, while Mandriva is now monkeying with pinit (and it looks quite promising), I prefer runit, which is similar to DJB’s daemontools. Now, the problem with runit is that it doesn’t use initscripts (sure, you could… I did it for qmail and djbdns running under Mandrake, but it was a PITA and rather annoying). Using native runit run scripts and runit itself running as init is much cleaner. Can you imagine Mandriva ever adopting this? Not in a million years. It’s not LSB-compatible, not even near anything the LSB would certify. So there was no point in bringing that up even though, for many things, runit is technically superior especially for running daemons.

I don’t like msec. I don’t like how msec does sneaky things behind your back. I don’t like how msec does anything, other than reporting. Do you think I could get that pulled out? Never. So I took msec, ripped out all of the “changing” stuff and left the reporting capabilities. Now that’s my idea of msec. Make it into Mandriva? No.

There’s a whole slew of stuff. I started, at the beginning, using a %build_opensls macro to wrap up my specific changes and try to make things compatible with Mandriva. But that was far too much work and left that bloat I wanted to get rid of in there. So I stopped trying to make it compatible. Annvix isn’t compatible with anything. It is entirely in a league of it’s own.

So it infuriates me when people (well, a certain someone actually) implied that I should be making an attempt to make an rpm macro compatible with Mandriva.. on the basis of compatibility. Like I want Annvix to be compatible with something. If Annvix is so far incompatible with pretty much anything (lack of initscripts, a different init, numerous other changes) why would I concern myself with using %_mkrel (which, BTW, is something that only Mandriva (main+contribs) and PLF use, AFAIK). I don’t even like %_mkrel, and yet I should be making it compatible? Yeah, that’ll work. I’ll use %_mkrel and you try to compile an Annvix package on Mandriva, install it, and see what happens. Oh, I know. A whole lot of nothing.

And besides, I don’t see why I need to make Annvix compatible with anything. I’m not marketting it. I’m not selling it. I’m not even actively advertising it. What am I doing with it? Hacking away and making something of interest to me. I honestly don’t care if it’s at all interesting to you, Joe, or the next guy. It makes no difference to me. In fact, it doesn’t matter if anyone uses Annvix at all. And if they do use it and want some help with it, well, they’ll just have to wait until time permits (unless they want to hire me as a consultant, of course). If they want some changes or something added, guess what? Unless I think it’s good for me they get to do it themselves or go home.

Let there be no illusion here. Annvix is 100% created for me. Obviously, I don’t mind if other people use it… it’s publically available after all. And I sincerely thank the few mirrors who are carrying it. And I do know of about 3-4 people who are using it and enjoy it. But (luckily for them) they pull their own weight. So them being a user is not a burden to me. The last thing Annvix will ever become is a burden, because I’ll never let it.

So, to all the Buchans of the world who seem to think I should make Annvix compatible with… something… let me tell you this. Annvix is not meant to be compatible with anything. It’s not LSB compliant. Heck, it may not even be a Linux distro forever… if I had the time, I’d fiddle with sticking the FreeBSD kernel in there and see how it panned out.

The sole point of Annvix is to provide something I can use on my own machines. It is to provide me with a learning experience (and trust me, developing your own distro will teach you stuff you never even thought about). It gives me an opportunity to play with stuff I couldn’t possibly play with on Mandriva unless I did some heavy hacking, and continued hacking because it could never be “mainline”. It’s not even close to being “mainline” enough to have conditional switches in there (which, BTW, I can’t stand).

And there’s a whole bunch of stuff about Mandriva (from a development standpoint) that I think is just plain old wrong.. but who am I to impose my point of view on others? Especially about some fairly radical ideas and practices? Nevermind the fact that I’m just the updates guy (which, I guess, means I’m not worth shit when it comes to R&D).

I guess the whole point of Annvix is it gives me something I can tinker with, quietly, in my own time, without having to fight and argue over every little thing that I want to do. It gives me complete freedom to do whatever I want, whether someone else thinks it’s good or bad, right or wrong. And, at the end of the day, isn’t that whole point of open source? Is the point of open source to be compatible with other stuff? Or is it to encourage the spread of new ideas, philosophies, and software? Is the point of open source for one guy who has absolutely nothing to do with Annvix to tell me what I should be doing with it? Since when was the use or abuse of open source software supposed to be controlled by dictators with absolutely no vested interest in a particular project? That would be like me telling, oh, I don’t know… Vixie how he should make BIND or cron work, despite the fact that I use djbdns and dcron and have absolutely no interest in using BIND or cron.

Does that make sense to you? Makes absolutely no sense to me. I’d be the biggest kind of asshole if I fired off an email to him telling that what he was doing was wrong because he didn’t make it compatible with something I’m using.

Well, that was longer than I expected. Hopefully if gives people some insight on my motives behind Annvix before they fire off their mouths pretending to know anything about what I’m about. Frankly, I don’t care if people think I’m doing things wrong with Annvix because I don’t care whether or not they use it. And if they like what I’m doing with Annvix and want to use it? Guess what! I don’t care about that either. It’s free for the taking, free for the raping, and free for the forking. Just don’t tell me what I should or should not be doing with it unless a) you’re using it and b) you’re prepared to pony up some time and effort to implement what you want done.

Oh, and as a quick bit of history, I did start Annvix (more or less) in September of 2003. The first “package” for Annvix was gcc (due to the SSP stuff) which Giuseppe implemented initially, and when I then built on as you can see with this first-ever changelog for Annvix (using the suffix “mdks” at that point for Mandrake Secure):

* Sun Sep 21 2003 Giuseppe Ghib?  3.3.1-2mdks
- Added Propolice Stack Protector and regenerated its patch (Patch300)

Thanks, Giuseppe! =)