Refresh Gpg Key

No worries my complaining about NixOS isn't done :). But for this blog post we take a break and talk complain about GPG. I use and love pass as my primary password manager. Which uses gpg to encrypt and decrypt files which are tracked via git. (I even have a small rofi script to access passwords quickly)

At some point my previous gpg key expired and since it was 10 years old it was time to update. A task that sounds easy enough:

gpg --full-generate-key
pass init $keyid

and that should be it. And it might be good enough depending on your gpg version and settings. Now the only step is to export this key and put it on my phone to unlock my password manager there. I use a combination of OpenKeychain: Easy PGP and Password Store.

Exporting and importing that via a USB stick on the phone was so easy.

gpg --export-secret-key $keyid > gpgprivate.key

Only to be then greeted with:

Encountered OpenPGP Exception during operation!

The new OpenPGP AEAD Mode

AEAD is Authenticated Encryption with Associated Data which is as far as i understand it a way to have unencrypted data (for example router header) as a part of your authenticated message. Meaning the receiver of the message can check if the header was modified.

And there are a bunch of incompatible modes / implementations for this. (See: for way more details) In summary OpenGPG defaults to a mode called OCB which is not standard and implementations like the Android App do not support.

The arch wiki contains a good description on how to disable AEAD on an existing key.

$ gpg --expert --edit-key <FINGERPRINT>
gpg> showpref
[ultimate] (1). Foobar McFooface (test) <>
    Cipher: AES256, AES192, AES, 3DES
    Digest: SHA512, SHA384, SHA256, SHA224, SHA1
    Compression: ZLIB, BZIP2, ZIP, Uncompressed
    Features: MDC, AEAD, Keyserver no-modify

gpg> setpref AES256 AES192 AES SHA512 SHA384 SHA256 SHA224 ZLIB BZIP2 ZIP
Set preference list to:
    Cipher: AES256, AES192, AES, 3DES
    Digest: SHA512, SHA384, SHA256, SHA224, SHA1
    Compression: ZLIB, BZIP2, ZIP, Uncompressed
    Features: MDC, Keyserver no-modify
Really update the preferences? (y/N) y


Now if we already updated the key in pass we unfortunately need to re-encrypt all files again.

for filename in ./*/*.gpg; do gpg -d -r $USER ./${filename} > ./${filename}.decrypt ; done
for filename in ./*/*.decrypt; do gpg -e -r $USER ./${filename} ; done
for filename in ./*/*.decrypt.gpg; do mv "${filename}" "${filename/.decrypt.gpg/}" ; done

git commit -am "re-encrypt passwords without AEAD mode"
git clean -dfx


And tada 🎉 our key works now on Android as well.

Build Custom Nixos Raspberry Pi Images

Since you might be not interested in me hating NixOS, Linux and the world in general i put my little rant at the end of this article. The first part is how to cross build a NixOS image for a Raspberry Pi 3 B+ from Fedora. I used compiling through binfmt QEMU. My Fedora laptop is a x86 system and we need to build a AArch64 image.

I assume that nix is already installed and binfmt is installed and works. And spoiler / warning no idea if that is proper or a good way to do it, it is just the way that worked for me.

$ nix --version
nix (Nix) 2.15.1

$ ls /proc/sys/fs/binfmt_misc/ | grep aarch64

$ systemctl status systemd-binfmt.service

We need to configure nix to use this. For this I added the following config to /etc/nix/nix.conf.

extra-platforms = aarch64-linux
extra-sandbox-paths = /usr/bin/qemu-aarch64-static

After that we need to restart the nix daemon.

$ systemctl restart nix-daemon.service

After that we are ready to create the config file:

$ cat configuration.sdImage.nix
{ config, pkgs, lib, ... }:
  nixpkgs.overlays = [
    (final: super: {
      makeModulesClosure = x:
        super.makeModulesClosure (x // { allowMissing = true; });
  system.stateVersion = lib.mkDefault "23.11";

  imports = [

  nixpkgs.hostPlatform.system = "aarch64-linux";
  sdImage.compressImage = false;

  # NixOS wants to enable GRUB by default
  boot.loader.grub.enable = false;
  # Enables the generation of /boot/extlinux/extlinux.conf
  boot.loader.generic-extlinux-compatible.enable = true;

  # Set to specific linux kernel version
  boot.kernelPackages = pkgs.linuxPackages_rpi3;

  # Needed for the virtual console to work on the RPi 3, as the default of 16M doesn't seem to be enough.
  # If behaves weirdly (I only saw the cursor) then try increasing this to 256M.
  # On a Raspberry Pi 4 with 4 GB, you should either disable this parameter or increase to at least 64M if you want the USB ports to work.
  boot.kernelParams = ["cma=256M"];

  # Settings
  # The rest of your config things

  # Use less privileged nixos user
  users.users.nixos = {
    isNormalUser = true;
    extraGroups = [ "wheel" "networkmanager" "video" ];
    # Allow the graphical user to login without password
    initialHashedPassword = "";

  # Allow the user to log in as root without a password.
  users.users.root.initialHashedPassword = "";

The overlays are quite important as there is some issue which I don't fully understand. If not added the error looks something like this where a kernel module was not found:

modprobe: FATAL: Module ahci not found in directory /nix/store/8bsagfwwxdvp9ybz37p092n131vnk8wz-linux-aarch64-unknown-linux-gnu-6.1.21-1.20230405-modules/lib/modules/6.1.21
error: builder for '/nix/store/jmb55l06cvdpvwwivny97aldzh147jwx-linux-aarch64-unknown-linux-gnu-6.1.21-1.20230405-modules-shrunk.drv' failed with exit code 1;
       last 3 log lines:
       > kernel version is 6.1.21
       > root module: ahci
       > modprobe: FATAL: Module ahci not found in directory /nix/store/8bsagfwwxdvp9ybz37p092n131vnk8wz-linux-aarch64-unknown-linux-gnu-6.1.21-1.20230405-modules/lib/modules/6.1.21
       For full logs, run 'nix log /nix/store/jmb55l06cvdpvwwivny97aldzh147jwx-linux-aarch64-unknown-linux-gnu-6.1.21-1.20230405-modules-shrunk.drv'.
error: 1 dependencies of derivation '/nix/store/' failed to build
error: 1 dependencies of derivation '/nix/store/j2gmvl3vaj083ww87lwfrnx81g6vias2-initrd-linux-aarch64-unknown-linux-gnu-6.1.21-1.20230405.drv' failed to build
building '/nix/store/vs0cg5kzbislprzrd3ya16n1xd532763-zfs-user-2.1.12-aarch64-unknown-linux-gnu.drv'...
error: 1 dependencies of derivation '/nix/store/gjhfjh9bb3ha0v03k7b4r3wvw4nxm7r3-nixos-system-aegaeon-23.11pre493358.a30520bf8ea.drv' failed to build
error: 1 dependencies of derivation '/nix/store/x5mnb1xfxk7kp0mbjw7ahxrz2yiv922s-ext4-fs.img-aarch64-unknown-linux-gnu.drv' failed to build
error: 1 dependencies of derivation '/nix/store/8qbjy9mnkrbyhj4kvl50m8ynzpgwmrpz-nixos-sd-image-23.11pre493358.a30520bf8ea-aarch64-linux.img-aarch64-unknown-linux-gnu.drv' failed to build

Don't forget to add your customization after # Settings. This is the place where you setup your user, enable required services, configure networking. In my case that's where most of the config is from this blog post: Build a simple dns with a Raspberry Pi and NixOS.

After that we can build (this takes some time!) and flash the image.

nix-build '<nixpkgs/nixos>' -A -I nixos-config=./configuration.sdImage.nix --option sandbox false --argstr system aarch64-linux


sudo -s
cat /path/to/img > /dev/sdX

The Rant

Why am I building a image myself instead of using the official image and just do what i have written in my earlier blog post Build a simple dns with a Raspberry Pi and NixOS. And the answer to that is part of my rant somehow NixOS is not able to upgrade / build on 23.11 on a Raspberry Pi it crashes for my either while downloading some packages or with some pid that either deadlocks or hangs for longer than i was willing to wait (more than 6 hours).

After I decided to try to cross build it was a real struggle to figure out how to do that. There are a lot of resources:

And a lot of them are not well structured or outdated. Which makes it very hard for a beginner like me to figure out where to start.

But with all this ranting i also want to point out that it seems like most NixOS user want to help you out. Thanks makefu for answering all my stupid NixOS questions and nova for pointing me to the correct github issue.

Incus Gitlab Runner

With the recent fork and drama around LXD it might be time to give Incus a chance.

Using Incus as GitLab runner is nice because it provides you with a simple interface to run containers and VMs for the cases where Docker is not enough. Helpfully there is a custom LXD GitLab runner provided by GitLab.

Based on that I created a custom Incus GitLab runner. Checkout:

This can be easily integrated into the deployment system used to setup GitLab runners. (Thinks Ansible)

It also assumes that you already installed Incus on the runner. To achieve that you can follow the official documentation for that. Or take some inspiration from the next section.

Installing Incus

Look at the official documentation. This is just a quick summary on how I did it.

mkdir -p /etc/apt/keyrings/
curl -fsSL -o /etc/apt/keyrings/zabbly.asc

sh -c 'cat <<EOF > /etc/apt/sources.list.d/zabbly-incus-stable.sources
Enabled: yes
Types: deb
Suites: $(. /etc/os-release && echo ${VERSION_CODENAME})
Components: main
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/zabbly.asc


apt-get update
apt-get install incus

sudo adduser ubuntu incus-admin

Big Kudos to zabbly and Stéphane Graber for providing pre-built images!

And to setup Incus I used cloud-config runcmd.

 - 'incus admin init --preseed < /etc/incus.seed && touch /etc/incus.init'

This assumes that you created your preseed config:

$ cat /etc/incus.seed
config: {}
- config:
    ipv4.address: auto
    ipv6.address: none
  description: ""
  name: incusbr0
  type: ""
  project: default
- config:
    size: 200GiB
  description: ""
  name: default
  driver: zfs
- config: {}
  description: ""
      name: eth0
      network: incusbr0
      type: nic
      path: /
      pool: default
      type: disk
  name: default
projects: []
cluster: null

Zfs Nixos

As I am still experimenting with my NixOS setup I thought it would be nice to separate the user-date onto a separate nvme ssd. The plan was to use ZFS and put my /var/lib on it. This would allow me to create snapshots which can be pushed or pulled to my other ZFS systems. That all sounded easy enough but took way longer than expected.


It all starts with a new NVME SSD. I got a WD Blue SN570 2000 GB, M.2 228 because it was very cheap. And here is my first learning apparently one should re-run nixos-generate-config or add the nvme module by hand to the hardware config (boot.initrd.availableKernelModules) to allow NixOS to correctly detect the new hardware. (I lost a lot of time to figure this out.)


Creating the ZFS pool is the usual. But one thing to note is the device name since NixOS imports using the /dev/disk/by-id/ path it is recommended to use that path to create the pool. The by-id name should also be consistent during hardware changes, while other mappings might change and lead to a broken pool. At least that is my understanding of it. (Source people on the internet, Inconsistent Device Names Across Reboot Cause Mount Failure Or Incorrect Mount in Linux)

sudo zpool create -f -O atime=off -O utf8only=on -O normalization=formD -O aclinherit=passthrough -O compression=zstd -O recordsize=1m -O exec=off tank /dev/disk/by-id/nvme-eui.e8238fa6bf530001001b448b4e246dab

Move data

On the new pool we create datasets and mount them.

zfs create tank/var -o canmount=on
zfs create tank/var/lib -o canmount=on

Then we can copy over all the current data from /var/lib.

# 1. stop all services accessing `/var/lib`
# 2. move data
sudo cp -r /var/lib/* /tank/var/lib/
sudo rm -rf /var/lib/
sudo zfs set mountpoint=/var/lib tank/var/lib

And here is the rest of my NixOS config for ZFS:

# Setup ZFS
# Offical resources:
# -
# -

# Enable support for ZSF and always use a compatible kernel
boot.supportedFilesystems = [ "zfs" ];
boot.zfs.forceImportRoot = false;
boot.kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages;

# head -c 8 /etc/machine-id
# The primary use case is to ensure when using ZFS 
# that a pool isn’t imported accidentally on a wrong machine.
networking.hostId = "aaaaaaaa";

# Enable scrubing once a week
services.zfs.autoScrub.enable = true;

# Names of the pools to import
boot.zfs.extraPools = [ "tank" ];

And in the end run sudo nixos-rebuild switch to build it and switch to the configuration.

Fucked up ZFS Pool

In the end I ended up doing everything again and starting fresh. Because my system did not import my ZFS pool after a reboot. Here are the key things i learned.

NixOS does import the pools by-id by running a command like this:

zpool import -d "/dev/disk/by-id" -N tank


And this can be configured via boot.zfs.devNodes source. Took a while to figure out since i usually just run zpool import tank.

And the behavior I saw was:

zpool import tank <- works
zpool import -d "/dev/disk/by-id" -N tank <- fails

As it turns out wipefs does not necessarily remove all zpool information from a disk.

$ sudo wipefs -a /dev/nvme0n1
/dev/nvme0n1: 8 bytes were erased at offset 0x1d1c10abc00 (zfs_member): 0c b1 ba 00 00 00 00 00
/dev/nvme0n1: 8 bytes were erased at offset 0x1d1c10a9800 (zfs_member): 0c b1 ba 00 00 00 00 00
/dev/nvme0n1: 8 bytes were erased at offset 0x1d1c10a8000 (zfs_member): 0c b1 ba 00 00 00 00 00

While wipefs reports everything deleted we can still check with zdb that there is in fact still a ZFS label on the disk.

$ sudo zdb -l /dev/nvme0n1
failed to unpack label 0
    version: 5000
    name: 'tank'
    state: 1
    txg: 47
    pool_guid: 16638860066397443734
    errata: 0
    hostid: 2138265770
    hostname: 'telesto'
    top_guid: 4799150557898763025
    guid: 4799150557898763025
    vdev_children: 1
        type: 'disk'
        id: 0
        guid: 4799150557898763025
        path: '/dev/disk/by-id/nvme-eui.e8238fa6bf530001001b448b4e246dab'
        whole_disk: 0
        metaslab_array: 64
        metaslab_shift: 34
        ashift: 9
        asize: 2000394125312
        is_log: 0
        create_txg: 4
    labels = 1 2 3

And the way to clear that is by dd-ing the right spots in the front and at the back of the disk.

sudo dd if=/dev/zero of=/dev/nvme0n1 count=4 bs=512k
sudo dd if=/dev/zero of=/dev/nvme0n1 oseek=3907027120

There is a superuser answer which shows how that works. And here is my lengthy back and forth where we figured out that this is the issue.

Other resources which where helpful

Dnsmasq On Nixos 2305

This is a small update on the evolved configuration from my Build a simple dns with a Raspberry Pi and NixOS blog post.

I upgraded to 23.05 and learned that i should run sudo nix-collect-garbage -d from time to time to avoid running out of disk space.

And here is the updated dnsmasq configuration:

networking.hostFiles = [(pkgs.fetchurl {
  url = "https://hostname.local/l33tname/hosts/raw/branch/main/hosts";
  sha256 = "14hsqsvc97xiqlrdmknj27krxm5l50p4nhafn7a23c365yxdhlbx";

services.dnsmasq.enable = true;
services.dnsmasq.alwaysKeepRunning = true;
services.dnsmasq.settings.server = [ "" "" "" ];
services.dnsmasq.settings = { cache-size = 500; };

As you can see with the latest version some config keys changed slightly. But the big new thing is that the hosts files is now fetched from my local git server. This allows me to version and edit this file in a singe place.

Note: The hash nix-prefetch-url $url should be updated if the file changes, otherwise NixOS will happily continue to use the the file fetched last time.