Making an anopa-powered bootable ISO
There is something I've been meaning to do for some time now, I really should have, yet haven't found some time to: writing some sort of proper/actual introduction to anopa.
So let's try to do this now, with an actual example. A little while back, I explained why I was looking for another init system, and since I couldn't find what I was looking for, I started working on my own. That's how anopa, an init system/service manager based on the supervision suite s6, came to be.
Introducing anopa
I like to think of anopa as a toolbox, a collection of tools and scripts aimed to provide what you need to build your own init system and service manager for Linux system, based around the supervision suite s6.
Notably, anopa provides some scripts that can be used for the different stages of the init process/PID1. That is to say, there's basically 3 stages:
Stage 1: booting up: prepare the system, mount file systems, etc
Stage 2: supervision: keep an eye an long-running processes/daemons and restart them should they fail. (Also reap orphaned zombies.)
Stage 3: shutting down: stop/kill everything and "undo" what was done during stage 1
s6-svscan is perfectly suited for and designed to be used as init (PID1) during phase 2. That leaves the other two up for anopa. At this point it would be a good idea to have a read of anopa to be a bit familiar with its concepts of services, dependencies and whatnot.
Done? Okay, good. So as you should know, the init process (PID 1) launched by the kernel is quite special, and cannot die. It has to exist from start to finish, but it can actually execute into another binary, which allows us to have separate tool for separate phases quite easily.
anopa provides a few execline scripts that can be used for this purpose. First
we have aa-stage1 which can be used as initial init process, and will simply
create the runtime repository /run/services using aa-enable(1), then exec
into s6-svscan; But not before having launched the aa-stage2 script.
As you can notice here, we're not quite sticking to the initial description of
the different stages. Our initial script didn't do much, and certainly didn't
prepare the system. This will be the job of aa-stage2 by calling
aa-start(1), leaving the actual stage 2 task up to s6-svscan, the new
PID1.
Some time later when shutting down the system, we'll simply instruct
s6-svscan to exec into aa-stage3, which will use aa-stop(1) to stop
all services. As a last step, it will pivot into /run/initramfs (putting the
old root filesystem into /root-fs) and exec into /shutdown with the same
argument it received from s6-svscan ("halt", "reboot" or "poweroff").
This last script's job is to make sure everything is down, umount the old root filesystem and properly do the expected action (e.g. trigger the reboot).
This could be done by e.g. having a oneshot service with only a stop script,
that populates /run/initramfs with what's needed to properly perform the last
operations, such as umounting /root-fs. In fact, aa-terminate(1) would be
pretty useful in such cases, to close/umount all that can be.
Or; -- A couple more scripts are actually provided by anopa, namely aa-stage0
and aa-stage4, which are intended to be used as init and shutdown inside
the initramfs, should you want to do that.
That way your initramfs works quite similarly to the actual system, with a call to aa-start(1) on boot to set thing up (though there can only be oneshots, obviously), and a matching call to aa-stop(1) when shutting down to properly shut down/close things.
This means the rootfs from the initramfs should be kept around, so we can
actually pivot back into it at the end of aa-stage3 -- then we can properly
stop everything. (Note that aa-stage4 actually does call aa-terminate(1)
after aa-stop(1) to handle the unexpected.)
Real-life example: a bootable ISO
As a way to show how a system can be set up and work using anopa, let's build a bootable ISO, using anopa in the initramfs as well. To make this work, this is what will need to happen:
In the initramfs, our init (aa-stage0) will be tasked to mount the root
filesystem. Since we'll likely need /dev, /proc and /sys to be mounted
so everything works fine, we'll do that, and right before being done - and
switching to the new root - we'll move them into /root-fs -- This mean they'll
already be mounted when the system's init (our aa-stage1) starts, so we won't
have to (re)do it then.
We will also mount /run and move it similarly, to make things simpler. For
instance, doing so will allow us to mount bind the rootfs (from the initramfs)
into /run/initramfs so we can pivot back into it when shutting down.
Now, mounting the root file system: This will be done by mounting the "boot
partition" where we'll find a root-fs.squashfs file. This file will then be
mounted (ro) on /run/squashfs, a tmpfs will be mounted on /run/overlay and
an overlay (overlayfs) will be mounted as /root-fs using the /run/squashfs
as lower end and /run/overlay as upper end, thus providing us with a rw file
system.
Again, mouting the "internals" into /run will allow us to get them back when
shutting down, so we can umount everything properly.
We'll then exec into the main init, our aa-stage1 script, which will start
aa-stage2 and exec into s6-svscan. The former will then start the actual
system initialization (via aa-start(1)), assuming all the virtual APIs are
already mounted, as they should be. And in this case, so will be /boot since
we had to in order to access our squashfs file, so we'll just move it (under
/root-fs) so that's already taken care of as well.
And as said before when shutting down, our aa-stage3 will stop all services
via aa-stop(1) and eventually pivot into /run/initramfs -- where
aa-stage4 will find back everything (notably in /run) so calling
aa-stop(1) can properly umount/close everything as needed (including
/boot)
Similar setup could be used on an "actual" system (as opposed to live ISO) to
e.g. luksOpen and mount the encrypted root fs during stage 0, and umount & close
during stage 4. (Though in such cases, /boot is likely to be (u)mounted in the
system, stages 1 & 3, not the initramfs.)
Unusual system configuration
anopa starts & stops services, and those services can be scripts or even binaries. It's all up to you, but in my case I'm going for execline scripts mostly, so that's what we'll use here.
Similarly, I've decided to make my systems, including this one, use an
"uncommon" scheme for some system configuration, notably there's no /etc/fstab
nor /etc/crypttab (because parsing), and instead things are set up via small
text files.
Specifically, definitions of file systems is done via files device,
mountpoint, fstype and optionally options to be found in
/etc/mount/$INSTANCE where $INSTANCE is meant to be a name, used as instance
of services such as mount@
For example, to have e.g. /dev/disk/by-label/foobar be mounted as ext4 in
/boot one would use a service mount@boot and set up a few files like so:
mkdir -p /etc/mount/boot
cd /etc/mount/boot
echo /dev/disk/by-label/foobar > device
echo ext4 > fstype
echo /boot > mountpoint
Similarly, for the use of encrypted devices a service crypt@ can be used, and
configuration done via files device and optionally type, name, offset
and keyfile in /etc/crypt/$INSTANCE (where $INSTANCE would usually be the
same as used for mount@, to make things clear & simple).
type must be a valid type for cryptsetup, defaulting to "luks". name must
contain the name to be used, else "luks-$INSTANCE" will be used. offset can
be used to specify an offset (i.e. --offset $OFFSET where $OFFSET is the
file's content, will be given to cryptsetup) Lastly keyfile must be the path
to the keyfile to be used (via --key-file option) else you'll be prompted for
a passphrase.
The offset option can be notably useful for encrypted swap partitions, when
you create a LUKS container first, then remove all keys; Point being to give
that partition a valid header and therefore a UUID, so it can be reliably used.
Then using an offset of 4040 allows to preserve said header (with type "plain"
and keyfile "/dev/urandom" as usual for encrypted swap partitions).
Again, none of this is part of anopa, but services I've made for my own needs,
and will be sharing & using here (well, mount@ at least, our ISO won't have
any encrypted devices). You could very well stick to using /etc/{fs,crypt}tab
with appropriate services instead, or do things completely differently, this is
merely an example of how things can be done.
Order, order, order
As mentioned elsewhere, anopa handles dependencies and will order services as
needed. Of course, "as needed" means as you've told it to do, via service
definitions. What I like to try to do is have most services be ordered against a
service sysinit.
This service is an empty one, i.e. a oneshot service with no scripts at all, and
is therefore only used for ordering other services. If all services have either
before/sysinit or after/sysinit then it can help ensuring a bunch of
essential services are run first, and then the rest. Just make sure to ask the
service (sysinit) to be started on boot, as ordering directives for unneeded
(i.e. not started/stopped) services are obviously ignored.
Creating an initramfs
Another thing that's not part of anopa per-se, but we'll need here, is creating
an initramfs. Being an Arch user, this will be done using mkinitcpio of
course.
What we'll do is replace the usual hooks such as "base" "keymap" or "udev" with "busybox" to install our busybox, "aa-base" to install the minimum required for things to work, and "aa-repo" which will create the runtime repository for the initramfs and add the needed configuration/binaries/modules.
This is also done by using a file mkinitcpio.hook that must exist in the
service definition directories, and can contain the 2 following functions:
hook_service: ran when the service is used. For services that are instantiated, ran only once.hook_instance: ran only for instantiated services, one per instance and with the instance name as first argument.
Both of those shall use the usual mkinitcpio's functions (add_binary, etc) to
fill the initramfs as needed. Should the service not need anything to be added
to the initramfs, an empty file will do (absence of the file will generate a
warning during initramfs creation).
mdev, not udev
As if to complicate things further, I'm also not gonna use udev. Similar reasons to why I changed init system have me wishing not to use udev anymore, and instead of going for one of the forks or similar projects, I've decided to have to figure out how all this works, and come up with my own helper scripts to handle things.
So that's what will happen here, with a service devd that connects to the
netlink interface (using s6-devd) and launches an helper program on uevents.
Said helper is no other that busybox's mdev which will be complemented with
some scripts of our own to e.g. create symlinks in
/dev/disk/by-{{uu,}id,label}
This is also why we're building our own busybox, because we need mdev (though not in the initramfs).
ISO maintenance
Now that you (hopefully) start to have an idea of what we're trying to do, let's see how it will actually be done. Truth be told, having a bootable ISO at hand is something I've always considered pretty useful, and that's why it was one of the first things I did after installing Arch Linux.
I did so using the archiso scripts and following instructions from the wiki,
and while it worked - I did make myself a bootable ISO with a live system
containing the few extra packages/configuration I wanted - it wasn't the easiest
procedure, I felt.
Not that it was especially complicated, but it had to do with writing a list of packages to install in a file, and to add extra files, they needed to be all in one place, then a script needed to be written; A script that would ran on boot, and whose purpose was to move files to the right places, set permissions, etc
In fact, to have a user account on the system, it also worked via a script creating said user on boot as well. I'm not sure why things were done this way, nor did I check whether things changed since then, but I know this is why I made myself a bootable ISO, and then that was it.
It wasn't maintained/updated, and until a few days ago when I decided to make my own anopa-powered ISO, that good old ISO from a few years back (and the pre-systemd era, in fact) was still in use.
All that to say, one of my goal was also to have a "better" system, something where I could easily work on, maintain, update the ISO without too much trouble.
How it works
So this is how it'll work: We'll need a folder with 2 things :
- a file
boot: "raw" block device, partitioned, containing our/bootfs. It will also have an MBR w/ syslinux installed, so we can use it as disk image when booting qemu, for testing purposes. - a folder
chroot: the root fs of the system, save for/bootThis is what we'll squash into ourroot-fs.squashfs
That way we can very simply work on the system: we mount the partition from
boot on chroot/boot and then we use pacman to install/update the system, we
can chroot in there and do whatever we need to, etc
We can also simply start a qemu VM using the boot file as disk to check
things, without the need to make the ISO file (though obviously making the
squashfs file is required).
3, 2, 1... go!
Alright, let's do it. As said before, I'm using Arch Linux, and this assumes you're running one as well. This first part is a basic Arch install procedure, feel free to adjust/skip as you wish (make sure to check the last bit, though, regarding boot loader installation).
First, let's create the boot file and set things up:
truncate -s 420M boot
cfdisk boot # and make one bootable partition
sudo losetup -P /dev/loop0 boot
sudo mkfs.ext4 -L THIS-LAND /dev/loop0p1
sudo mkdir -p chroot/boot
sudo mount -B chroot chroot
sudo mount /dev/loop0p1 chroot/boot
So we will have to use a label, obviously, since that's all we'll have when
booting from a CD to identify our /boot partition. I've decided to call mine
"THIS-LAND" because why not. Of course this isn't right per ISO9660 specs
(because of the dash) and we'll get a warning when making the ISO. I decided I
didn't care, feel free to use any other label, even one compatible with the
specs.
You'll note we mount chroot on itself, this is so later on the root of the chroot is seen as a mountpoint, helps e.g. with mkinitcpio.
Now on to create some obligatory directories:
sudo mkdir -m 0755 -p chroot/var/{cache/pacman/pkg,lib/pacman,log} chroot/{dev,run,etc}
sudo mkdir -m 1777 -p chroot/tmp
sudo mkdir -m 0555 -p chroot/{sys,proc}
Mount the apis and let's install some "base system":
sudo mount -B /dev chroot/dev
sudo mount -B /proc chroot/proc
sudo mount -B /sys chroot/sys
sudo mount -t tmpfs -o mode=0755,nosuid,nodev tmpfs chroot/run
sudo mount -t tmpfs -o mode=1777,nosuid tmpfs chroot/tmp
sudo pacman -r chroot -Sy bash bzip2 coreutils cryptsetup device-mapper \
diffutils e2fsprogs file filesystem findutils gawk gcc-libs gettext \
glibc grep gzip inetutils iproute2 iputils less licenses linux \
logrotate man-db man-pages pacman pciutils perl procps-ng psmisc sed \
shadow sysfsutils tar texinfo util-linux which
This selection of packages is mine, feel free to use one that fits you best, or just go for the whole "base" group even. Note that this isn't a minimum required, e.g. it includes cryptsetup which isn't required for what we're doing. But my systems are encrypted, so I need to be able to access them from the ISO.
In typical install trope, let's copy keyring & mirrorlist:
sudo cp -a /etc/pacman.d/gnupg chroot/etc/pacman.d
sudo cp -a /etc/pacman.d/mirrorlist chroot/etc/pacman.d
Time to set up some basic config now - feel free to adjust to your needs ofc :
sudo sed -ie 's/^#\(en_US.UTF-8\)/\1/' chroot/etc/locale.gen
sudo chroot chroot locale-gen
echo LANG=en_US.utf8 | sudo tee chroot/etc/locale.conf
echo KEYMAP=fr-latin1 | sudo tee chroot/etc/vconsole.conf
sudo ln -s /usr/share/zoneinfo/Europe/Paris chroot/etc/localtime
echo this-land | sudo tee chroot/etc/hostname
And install our bootloader, since as mentioned earlier we want the boot file
to be usable as disk in a VM:
sudo pacman -r chroot -S syslinux
sudo chroot chroot sh
cp -r /lib/syslinux/bios/*.c32 /boot/syslinux/ # or just those needed
extlinux -i /boot/syslinux
dd if=/lib/syslinux/bios/mbr.bin of=/dev/loop0 bs=440 count=1
touch /boot/syslinux/SYSLINUX_AUTOUPDATE # for "auto-update"
cp /lib/syslinux/bios/isolinux.bin /boot # for xorrisofs
exit
vim chroot/boot/syslinux/syslinux.cfg # APPEND init=/lib/anopa/aa-stage1
Now it gets specific
At this point we've installed a basic system under chroot -- Time to dive into
our own init system and whatnot.
It starts with a few packages:
sudo pacman -r chroot -S binutils execline-musl-git busybox-musl-jjk \
s6-musl-git s6-devd-musl-git anopa-musl-git
I'm building my own busybox so it can be statically linked to musl, and also so it includes mdev since we'll need it (remember: no udev). Other packages are also built statically against musl, though you could do it differently; All PKGBUILDs can be found on my github, or in the AUR (Noting that I don't maintain any s6-* packages there, not that it should matter).
s6-devd is a small package containing only the s6-devd tool I use to connect
to the netlink interface and process uevents (or, send them to mdev). You could
also just use s6-linux-utils if you need/want the other tools.
And of course if you intend to use udev, you don't need it nor to get mdev in busybox.
Now, to help speed things up, I've made a few extra packages. First off is the mdev configuration & helpers, then some of those aforementioned services.
sudo pacman -r chroot -S lila-mdev lila-system-services lila-services
Relevant PKGBUILDs can be found on github as usual. lila-services should only
contain "generic" servicedirs that relate to specific packages/daemons (e.g.
getty, consolekit, dbus, openvpn, nginx, etc), whereas lila-system-services
contains mostly oneshot services for system initialization, and those can be
pretty custom made. This is for example where the previously mentionned mount@
would be found, as well as a few others. It also contains the mkinitcpio hooks
we'll use to build our anopa-based initramfs.
Feel free to look into those and see what services look like. As said earlier, most of them are execline scripts, all are examples - you can use them, or not.
It should be mentioned that those of course go with s6-devd+mdev and no udev,
but also that there's a log-events service, which comes with a few others it
needs (such as fdholdingd), which are all meant to allow to have loggers send
some data to this log-events service (via writing to their stdout, connected
to the log-events fifo), as done in the default logger log.
This relies not only on using s6-log as logger, but a patched version at that.
(Patch is here) to include prefix when writing on stdout. It might not be of use
to everyone, it certainly won't be for us here, more on that in a minute.
Configuration time
Now to keep helping things, I've also made a couple of tarballs that will come in handy. First off is lila-lazy which contains a bunch of configuration that I basically use as base on every anopa-based system I build.
This notably includes everything that should be needed for the initramfs, and
most things for a basic system init. Afterwards, you'll usually only need to go
edit a couple of files - /etc/mount/{boot,root-fs}/device - to set them up
properly. (Maybe also fill things in /etc/crypt/ if you were to use encrypted
devices.)
We'll need to make some adjustments from that, and the services from
lila-system-services as well.
Indeed, if our case things are a bit different, and slightly more complicated when it comes to the initramfs, since we have quite a few specific things to do there. So here comes lila-this-land which contains all the adjustments we need to get our configuration ready.
This includes new files, updated files, and a few files whose name starts with a
dash, indicating that the original file must be removed. Those are located in
etc/anopa/{enabled,listdirs/onboot} and require manual attention. As I said,
a file starting with a dash means both it and its no-dash counterpart shall be
removed. There's also a special case with +kmsgd, which means the folder (from
lila-lazy) should be removed, and the (empty) file renamed to kmsgd
So extract both tarballs in chroot then take care of the few files to remove
(and the kmsgd case), and that's it. Pretty simple, uh?
You'll notice some extra services, e.g. to mount the squashfs and the overlay.
There's also mdev-sr which is used to run our mdev helper that creates
symlinks on the CD-ROM, so if we're actually booting from a CD, we can find
/boot with our usual symlink in /dev/disk/by-label.
A service boot-fstype is there to get the fstype of said device, before we
mount it on /boot, and update the /etc/mount/boot/fstype file. This is
because while it would be iso9660 when booting a CD/ISO, when using the boot
file as virtual disk, it will be ext4. And because mounting happens with
aa-mount which doesn't have an "auto" mode, we need to do this.
Another extra service is memdisk and it is there to support the case of
booting the ISO file from syslinux. Because that's actually how I'll use the
final ISO in the end myself: booting it from syslinux, via memdisk. And in that
case, we need this service to ensure we can find, and mount, /boot
With all that done, a few extra directories need to be created, for our logging system to work:
sudo chroot chroot
mkdir /var/log/{boot,dbus,devd,kmsgd,syslogd,uncaught-logs}
useradd -r -g log -s /bin/nologin -d /var/log log
chown log:log /var/log/{boot,dbus,devd,kmsgd,syslogd,uncaught-logs}
exit
After that, we're almost done.
mkinitcpio time
Only thing left, to get the system ready at least, is actually build the
initramfs. So we need to edit mkinitcpio.conf to set proper hooks, e.g:
HOOKS="busybox aa-base aa-repo block filesystems keyboard"
We can also remove the 'fallback' from linux.preset (since we don't use
autodetect), rm chroot/boot/initramfs-linux-fallback.img, then rebuild; and
that's our anopa-powered initramfs.
Squashing...
The root filesystem is now ready. Of course, for this whole thing to work, we
need to squash it into a file, so let's do just that, making sure to exclude
/boot of course:
sudo mksquashfs chroot root-fs.squashfs -noappend -wildcards -e boot/'*'
sudo mv -f root-fs.squashfs chroot/boot
Obviously, we should have umounted the apis before squashing:
sudo umount chroot/{dev,proc,sys,run,tmp}
It boots!
At this point, it should be ready to boot. And using qemu and the boot file as
hdd, it should indeed do the trick:
qemu-system-x86_64 -m 1G -drive file=boot,index=0,media=disk,format=raw
Now all that's left is actually making an ISO.
Finally, an ISO is brought into the world.
This final step is actually pretty simple, all it takes is run xorrisofs with
the proper arguments. Make sure to set -volid to whatever the label of your
/boot fs is, otherwise expect breakage.
sudo xorrisofs -output this-land.iso -volid THIS-LAND \
-iso-level 2 -allow-lowercase -omit-period \
-eltorito-boot isolinux.bin -eltorito-catalog boot.cat \
-no-emul-boot -boot-load-size 4 -boot-info-table \
-isohybrid-mbr chroot/lib/syslinux/bios/isohdpfx.bin \
chroot/boot
Voilà! We now have a bootable ISO, with a system using anopa as init system as well as in the initramfs.