Making an anopa-powered bootable ISO

There is something I've been meaning to do for some time now, I really should have, yet haven't found some time to: writing some sort of proper/actual introduction to anopa.

So let's try to do this now, with an actual example. A little while back, I explained why I was looking for another init system, and since I couldn't find what I was looking for, I started working on my own. That's how anopa, an init system/service manager based on the supervision suite s6, came to be.

Introducing anopa

I like to think of anopa as a toolbox, a collection of tools and scripts aimed to provide what you need to build your own init system and service manager for Linux system, based around the supervision suite s6.

Notably, anopa provides some scripts that can be used for the different stages of the init process/PID1. That is to say, there's basically 3 stages:

  • Stage 1: booting up: prepare the system, mount file systems, etc

  • Stage 2: supervision: keep an eye an long-running processes/daemons and restart them should they fail. (Also reap orphaned zombies.)

  • Stage 3: shutting down: stop/kill everything and "undo" what was done during stage 1

s6-svscan is perfectly suited for and designed to be used as init (PID1) during phase 2. That leaves the other two up for anopa. At this point it would be a good idea to have a read of anopa to be a bit familiar with its concepts of services, dependencies and whatnot.

Done? Okay, good. So as you should know, the init process (PID 1) launched by the kernel is quite special, and cannot die. It has to exist from start to finish, but it can actually execute into another binary, which allows us to have separate tool for separate phases quite easily.

anopa provides a few execline scripts that can be used for this purpose. First we have aa-stage1 which can be used as initial init process, and will simply create the runtime repository /run/services using aa-enable(1), then exec into s6-svscan; But not before having launched the aa-stage2 script.

As you can notice here, we're not quite sticking to the initial description of the different stages. Our initial script didn't do much, and certainly didn't prepare the system. This will be the job of aa-stage2 by calling aa-start(1), leaving the actual stage 2 task up to s6-svscan, the new PID1.

Some time later when shutting down the system, we'll simply instruct s6-svscan to exec into aa-stage3, which will use aa-stop(1) to stop all services. As a last step, it will pivot into /run/initramfs (putting the old root filesystem into /root-fs) and exec into /shutdown with the same argument it received from s6-svscan ("halt", "reboot" or "poweroff").

This last script's job is to make sure everything is down, umount the old root filesystem and properly do the expected action (e.g. trigger the reboot).

This could be done by e.g. having a oneshot service with only a stop script, that populates /run/initramfs with what's needed to properly perform the last operations, such as umounting /root-fs. In fact, aa-terminate(1) would be pretty useful in such cases, to close/umount all that can be.

Or; -- A couple more scripts are actually provided by anopa, namely aa-stage0 and aa-stage4, which are intended to be used as init and shutdown inside the initramfs, should you want to do that.

That way your initramfs works quite similarly to the actual system, with a call to aa-start(1) on boot to set thing up (though there can only be oneshots, obviously), and a matching call to aa-stop(1) when shutting down to properly shut down/close things.

This means the rootfs from the initramfs should be kept around, so we can actually pivot back into it at the end of aa-stage3 -- then we can properly stop everything. (Note that aa-stage4 actually does call aa-terminate(1) after aa-stop(1) to handle the unexpected.)

Real-life example: a bootable ISO

As a way to show how a system can be set up and work using anopa, let's build a bootable ISO, using anopa in the initramfs as well. To make this work, this is what will need to happen:

In the initramfs, our init (aa-stage0) will be tasked to mount the root filesystem. Since we'll likely need /dev, /proc and /sys to be mounted so everything works fine, we'll do that, and right before being done - and switching to the new root - we'll move them into /root-fs -- This mean they'll already be mounted when the system's init (our aa-stage1) starts, so we won't have to (re)do it then.

We will also mount /run and move it similarly, to make things simpler. For instance, doing so will allow us to mount bind the rootfs (from the initramfs) into /run/initramfs so we can pivot back into it when shutting down.

Now, mounting the root file system: This will be done by mounting the "boot partition" where we'll find a root-fs.squashfs file. This file will then be mounted (ro) on /run/squashfs, a tmpfs will be mounted on /run/overlay and an overlay (overlayfs) will be mounted as /root-fs using the /run/squashfs as lower end and /run/overlay as upper end, thus providing us with a rw file system.

Again, mouting the "internals" into /run will allow us to get them back when shutting down, so we can umount everything properly.

We'll then exec into the main init, our aa-stage1 script, which will start aa-stage2 and exec into s6-svscan. The former will then start the actual system initialization (via aa-start(1)), assuming all the virtual APIs are already mounted, as they should be. And in this case, so will be /boot since we had to in order to access our squashfs file, so we'll just move it (under /root-fs) so that's already taken care of as well.

And as said before when shutting down, our aa-stage3 will stop all services via aa-stop(1) and eventually pivot into /run/initramfs -- where aa-stage4 will find back everything (notably in /run) so calling aa-stop(1) can properly umount/close everything as needed (including /boot)

Similar setup could be used on an "actual" system (as opposed to live ISO) to e.g. luksOpen and mount the encrypted root fs during stage 0, and umount & close during stage 4. (Though in such cases, /boot is likely to be (u)mounted in the system, stages 1 & 3, not the initramfs.)

Unusual system configuration

anopa starts & stops services, and those services can be scripts or even binaries. It's all up to you, but in my case I'm going for execline scripts mostly, so that's what we'll use here.

Similarly, I've decided to make my systems, including this one, use an "uncommon" scheme for some system configuration, notably there's no /etc/fstab nor /etc/crypttab (because parsing), and instead things are set up via small text files.

Specifically, definitions of file systems is done via files device, mountpoint, fstype and optionally options to be found in /etc/mount/$INSTANCE where $INSTANCE is meant to be a name, used as instance of services such as mount@

For example, to have e.g. /dev/disk/by-label/foobar be mounted as ext4 in /boot one would use a service mount@boot and set up a few files like so:

mkdir -p /etc/mount/boot
cd /etc/mount/boot
echo /dev/disk/by-label/foobar > device
echo ext4 > fstype
echo /boot > mountpoint

Similarly, for the use of encrypted devices a service crypt@ can be used, and configuration done via files device and optionally type, name, offset and keyfile in /etc/crypt/$INSTANCE (where $INSTANCE would usually be the same as used for mount@, to make things clear & simple).

type must be a valid type for cryptsetup, defaulting to "luks". name must contain the name to be used, else "luks-$INSTANCE" will be used. offset can be used to specify an offset (i.e. --offset $OFFSET where $OFFSET is the file's content, will be given to cryptsetup) Lastly keyfile must be the path to the keyfile to be used (via --key-file option) else you'll be prompted for a passphrase.

The offset option can be notably useful for encrypted swap partitions, when you create a LUKS container first, then remove all keys; Point being to give that partition a valid header and therefore a UUID, so it can be reliably used. Then using an offset of 4040 allows to preserve said header (with type "plain" and keyfile "/dev/urandom" as usual for encrypted swap partitions).

Again, none of this is part of anopa, but services I've made for my own needs, and will be sharing & using here (well, mount@ at least, our ISO won't have any encrypted devices). You could very well stick to using /etc/{fs,crypt}tab with appropriate services instead, or do things completely differently, this is merely an example of how things can be done.

Order, order, order

As mentioned elsewhere, anopa handles dependencies and will order services as needed. Of course, "as needed" means as you've told it to do, via service definitions. What I like to try to do is have most services be ordered against a service sysinit.

This service is an empty one, i.e. a oneshot service with no scripts at all, and is therefore only used for ordering other services. If all services have either before/sysinit or after/sysinit then it can help ensuring a bunch of essential services are run first, and then the rest. Just make sure to ask the service (sysinit) to be started on boot, as ordering directives for unneeded (i.e. not started/stopped) services are obviously ignored.

Creating an initramfs

Another thing that's not part of anopa per-se, but we'll need here, is creating an initramfs. Being an Arch user, this will be done using mkinitcpio of course.

What we'll do is replace the usual hooks such as "base" "keymap" or "udev" with "busybox" to install our busybox, "aa-base" to install the minimum required for things to work, and "aa-repo" which will create the runtime repository for the initramfs and add the needed configuration/binaries/modules.

This is also done by using a file mkinitcpio.hook that must exist in the service definition directories, and can contain the 2 following functions:

  • hook_service : ran when the service is used. For services that are instantiated, ran only once.

  • hook_instance : ran only for instantiated services, one per instance and with the instance name as first argument.

Both of those shall use the usual mkinitcpio's functions (add_binary, etc) to fill the initramfs as needed. Should the service not need anything to be added to the initramfs, an empty file will do (absence of the file will generate a warning during initramfs creation).

mdev, not udev

As if to complicate things further, I'm also not gonna use udev. Similar reasons to why I changed init system have me wishing not to use udev anymore, and instead of going for one of the forks or similar projects, I've decided to have to figure out how all this works, and come up with my own helper scripts to handle things.

So that's what will happen here, with a service devd that connects to the netlink interface (using s6-devd) and launches an helper program on uevents. Said helper is no other that busybox's mdev which will be complemented with some scripts of our own to e.g. create symlinks in /dev/disk/by-{{uu,}id,label}

This is also why we're building our own busybox, because we need mdev (though not in the initramfs).

ISO maintenance

Now that you (hopefully) start to have an idea of what we're trying to do, let's see how it will actually be done. Truth be told, having a bootable ISO at hand is something I've always considered pretty useful, and that's why it was one of the first things I did after installing Arch Linux.

I did so using the archiso scripts and following instructions from the wiki, and while it worked - I did make myself a bootable ISO with a live system containing the few extra packages/configuration I wanted - it wasn't the easiest procedure, I felt.

Not that it was especially complicated, but it had to do with writing a list of packages to install in a file, and to add extra files, they needed to be all in one place, then a script needed to be written; A script that would ran on boot, and whose purpose was to move files to the right places, set permissions, etc

In fact, to have a user account on the system, it also worked via a script creating said user on boot as well. I'm not sure why things were done this way, nor did I check whether things changed since then, but I know this is why I made myself a bootable ISO, and then that was it.

It wasn't maintained/updated, and until a few days ago when I decided to make my own anopa-powered ISO, that good old ISO from a few years back (and the pre-systemd era, in fact) was still in use.

All that to say, one of my goal was also to have a "better" system, something where I could easily work on, maintain, update the ISO without too much trouble.

How it works

So this is how it'll work: We'll need a folder with 2 things :

  • a file boot : "raw" block device, partitioned, containing our /boot fs. It will also have an MBR w/ syslinux installed, so we can use it as disk image when booting qemu, for testing purposes.
  • a folder chroot : the root fs of the system, save for /boot This is what we'll squash into our root-fs.squashfs

That way we can very simply work on the system: we mount the partition from boot on chroot/boot and then we use pacman to install/update the system, we can chroot in there and do whatever we need to, etc

We can also simply start a qemu VM using the boot file as disk to check things, without the need to make the ISO file (though obviously making the squashfs file is required).

3, 2, 1... go!

Alright, let's do it. As said before, I'm using Arch Linux, and this assumes you're running one as well. This first part is a basic Arch install procedure, feel free to adjust/skip as you wish (make sure to check the last bit, though, regarding boot loader installation).

First, let's create the boot file and set things up:

truncate -s 420M boot
cfdisk boot # and make one bootable partition
sudo losetup -P /dev/loop0 boot
sudo mkfs.ext4 -L THIS-LAND /dev/loop0p1
sudo mkdir -p chroot/boot
sudo mount -B chroot chroot
sudo mount /dev/loop0p1 chroot/boot

So we will have to use a label, obviously, since that's all we'll have when booting from a CD to identify our /boot partition. I've decided to call mine "THIS-LAND" because why not. Of course this isn't right per ISO9660 specs (because of the dash) and we'll get a warning when making the ISO. I decided I didn't care, feel free to use any other label, even one compatible with the specs.

You'll note we mount chroot on itself, this is so later on the root of the chroot is seen as a mountpoint, helps e.g. with mkinitcpio.

Now on to create some obligatory directories:

sudo mkdir -m 0755 -p chroot/var/{cache/pacman/pkg,lib/pacman,log} chroot/{dev,run,etc}
sudo mkdir -m 1777 -p chroot/tmp
sudo mkdir -m 0555 -p chroot/{sys,proc}

Mount the apis and let's install some "base system":

sudo mount -B /dev chroot/dev
sudo mount -B /proc chroot/proc
sudo mount -B /sys chroot/sys
sudo mount -t tmpfs -o mode=0755,nosuid,nodev tmpfs chroot/run
sudo mount -t tmpfs -o mode=1777,nosuid tmpfs chroot/tmp
sudo pacman -r chroot -Sy bash bzip2 coreutils cryptsetup device-mapper \
    diffutils e2fsprogs file filesystem findutils gawk gcc-libs gettext \
    glibc grep gzip inetutils iproute2 iputils less licenses linux \
    logrotate man-db man-pages pacman pciutils perl procps-ng psmisc sed \
    shadow sysfsutils tar texinfo util-linux which

This selection of packages is mine, feel free to use one that fits you best, or just go for the whole "base" group even. Note that this isn't a minimum required, e.g. it includes cryptsetup which isn't required for what we're doing. But my systems are encrypted, so I need to be able to access them from the ISO.

In typical install trope, let's copy keyring & mirrorlist:

sudo cp -a /etc/pacman.d/gnupg chroot/etc/pacman.d
sudo cp -a /etc/pacman.d/mirrorlist chroot/etc/pacman.d

Time to set up some basic config now - feel free to adjust to your needs ofc :

sudo sed -ie 's/^#\(en_US.UTF-8\)/\1/' chroot/etc/locale.gen
sudo chroot chroot locale-gen
echo LANG=en_US.utf8 | sudo tee chroot/etc/locale.conf
echo KEYMAP=fr-latin1 | sudo tee chroot/etc/vconsole.conf
sudo ln -s /usr/share/zoneinfo/Europe/Paris chroot/etc/localtime
echo this-land | sudo tee chroot/etc/hostname

And install our bootloader, since as mentioned earlier we want the boot file to be usable as disk in a VM:

sudo pacman -r chroot -S syslinux
sudo chroot chroot sh
    cp -r /lib/syslinux/bios/*.c32 /boot/syslinux/ # or just those needed
    extlinux -i /boot/syslinux
    dd if=/lib/syslinux/bios/mbr.bin of=/dev/loop0 bs=440 count=1
    touch /boot/syslinux/SYSLINUX_AUTOUPDATE # for "auto-update"
    cp /lib/syslinux/bios/isolinux.bin /boot # for xorrisofs
    exit
vim chroot/boot/syslinux/syslinux.cfg # APPEND init=/lib/anopa/aa-stage1

Now it gets specific

At this point we've installed a basic system under chroot -- Time to dive into our own init system and whatnot.

It starts with a few packages:

sudo pacman -r chroot -S binutils execline-musl-git busybox-musl-jjk \
    s6-musl-git s6-devd-musl-git anopa-musl-git

I'm building my own busybox so it can be statically linked to musl, and also so it includes mdev since we'll need it (remember: no udev). Other packages are also built statically against musl, though you could do it differently; All PKGBUILDs can be found on my github, or in the AUR (Noting that I don't maintain any s6-* packages there, not that it should matter).

s6-devd is a small package containing only the s6-devd tool I use to connect to the netlink interface and process uevents (or, send them to mdev). You could also just use s6-linux-utils if you need/want the other tools.

And of course if you intend to use udev, you don't need it nor to get mdev in busybox.

Now, to help speed things up, I've made a few extra packages. First off is the mdev configuration & helpers, then some of those aforementioned services.

sudo pacman -r chroot -S lila-mdev lila-system-services lila-services

Relevant PKGBUILDs can be found on github as usual. lila-services should only contain "generic" servicedirs that relate to specific packages/daemons (e.g. getty, consolekit, dbus, openvpn, nginx, etc), whereas lila-system-services contains mostly oneshot services for system initialization, and those can be pretty custom made. This is for example where the previously mentionned mount@ would be found, as well as a few others. It also contains the mkinitcpio hooks we'll use to build our anopa-based initramfs.

Feel free to look into those and see what services look like. As said earlier, most of them are execline scripts, all are examples - you can use them, or not.

It should be mentioned that those of course go with s6-devd+mdev and no udev, but also that there's a log-events service, which comes with a few others it needs (such as fdholdingd), which are all meant to allow to have loggers send some data to this log-events service (via writing to their stdout, connected to the log-events fifo), as done in the default logger log.

This relies not only on using s6-log as logger, but a patched version at that. (Patch is here) to include prefix when writing on stdout. It might not be of use to everyone, it certainly won't be for us here, more on that in a minute.

Configuration time

Now to keep helping things, I've also made a couple of tarballs that will come in handy. First off is lila-lazy which contains a bunch of configuration that I basically use as base on every anopa-based system I build.

This notably includes everything that should be needed for the initramfs, and most things for a basic system init. Afterwards, you'll usually only need to go edit a couple of files - /etc/mount/{boot,root-fs}/device - to set them up properly. (Maybe also fill things in /etc/crypt/ if you were to use encrypted devices.)

We'll need to make some adjustments from that, and the services from lila-system-services as well.

Indeed, if our case things are a bit different, and slightly more complicated when it comes to the initramfs, since we have quite a few specific things to do there. So here comes lila-this-land which contains all the adjustments we need to get our configuration ready.

This includes new files, updated files, and a few files whose name starts with a dash, indicating that the original file must be removed. Those are located in etc/anopa/{enabled,listdirs/onboot} and require manual attention. As I said, a file starting with a dash means both it and its no-dash counterpart shall be removed. There's also a special case with +kmsgd, which means the folder (from lila-lazy) should be removed, and the (empty) file renamed to kmsgd

So extract both tarballs in chroot then take care of the few files to remove (and the kmsgd case), and that's it. Pretty simple, uh?

You'll notice some extra services, e.g. to mount the squashfs and the overlay. There's also mdev-sr which is used to run our mdev helper that creates symlinks on the CD-ROM, so if we're actually booting from a CD, we can find /boot with our usual symlink in /dev/disk/by-label.

A service boot-fstype is there to get the fstype of said device, before we mount it on /boot, and update the /etc/mount/boot/fstype file. This is because while it would be iso9660 when booting a CD/ISO, when using the boot file as virtual disk, it will be ext4. And because mounting happens with aa-mount which doesn't have an "auto" mode, we need to do this.

Another extra service is memdisk and it is there to support the case of booting the ISO file from syslinux. Because that's actually how I'll use the final ISO in the end myself: booting it from syslinux, via memdisk. And in that case, we need this service to ensure we can find, and mount, /boot

With all that done, a few extra directories need to be created, for our logging system to work:

sudo chroot chroot
    mkdir /var/log/{boot,dbus,devd,kmsgd,syslogd,uncaught-logs}
    useradd -r -g log -s /bin/nologin -d /var/log log
    chown log:log /var/log/{boot,dbus,devd,kmsgd,syslogd,uncaught-logs}
    exit

After that, we're almost done.

mkinitcpio time

Only thing left, to get the system ready at least, is actually build the initramfs. So we need to edit mkinitcpio.conf to set proper hooks, e.g:

HOOKS="busybox aa-base aa-repo block filesystems keyboard"

We can also remove the 'fallback' from linux.preset (since we don't use autodetect), rm chroot/boot/initramfs-linux-fallback.img, then rebuild; and that's our anopa-powered initramfs.

Squashing...

The root filesystem is now ready. Of course, for this whole thing to work, we need to squash it into a file, so let's do just that, making sure to exclude /boot of course:

sudo mksquashfs chroot root-fs.squashfs -noappend -wildcards -e boot/'*'
sudo mv -f root-fs.squashfs chroot/boot

Obviously, we should have umounted the apis before squashing:

sudo umount chroot/{dev,proc,sys,run,tmp}

It boots!

At this point, it should be ready to boot. And using qemu and the boot file as hdd, it should indeed do the trick:

qemu-system-x86_64 -m 1G -drive file=boot,index=0,media=disk,format=raw

Now all that's left is actually making an ISO.

Finally, an ISO is brought into the world.

This final step is actually pretty simple, all it takes is run xorrisofs with the proper arguments. Make sure to set -volid to whatever the label of your /boot fs is, otherwise expect breakage.

sudo xorrisofs -output this-land.iso -volid THIS-LAND \
    -iso-level 2 -allow-lowercase -omit-period \
    -eltorito-boot isolinux.bin -eltorito-catalog boot.cat \
    -no-emul-boot -boot-load-size 4 -boot-info-table \
    -isohybrid-mbr chroot/lib/syslinux/bios/isohdpfx.bin \
    chroot/boot

Voilà! We now have a bootable ISO, with a system using anopa as init system as well as in the initramfs.

Top of Page