Home on ./concord.sh

Blog

Sun, 17 Nov 2019 14:42:54 -0500

My personal blog.

Links

Wed, 30 Jun 2021 21:52:25 -0400

madaidan’s Insecurities

https://madaidans-insecurities.github.io/

madaidan is a security researcher most known for his contributions to the Whonix project. He is a knowledgeable individual who maintains an accurate and detailed collection of guides and bulletins on various privacy and security topics on his site.

GrapheneOS

https://grapheneos.org/

GrapheneOS is a third-party Android operating system with a focus on privacy and security. It employs a variety of different technologies and techniques to strengthen defenses against exploitation, and has contributed much of this work back upstream to the Android Open Source Project.

About

Sun, 17 Nov 2019 12:16:26 -0500

I’m 0xC0ncord, or just “Concord,” and this site is a place where I talk to myself. I write about various information security topics. I contribute to open source projects such as the SELinux reference policy and work in systems administration professionally. This site is the work of a personal motivation effort to share as much useful information I gather publicly, in the hopes that it will be useful. Most of my work is generally geared towards security research on the personal Linux desktop.

I can be contacted on a few various platforms, namely Matrix. I am one of the operators of a well-known Matrix room, but can also be reached directly.

This site and its theme are open source!

You are viewing this site at commit 3b256f1 and the theme at commit 3eedb91.

Writing a Kubernetes Operator to Fix IPv6

Sun, 27 Apr 2025 11:37:09 -0400

I have an abusive relationship with IPv6.

Admittedly, I’m a bit of a stickler for “doing things the right way,” and part of that includes making sure my self-hosted apps are accessible over an IPv6-only network. In today’s age that means using dual-stack networking.

Unfortunately, it seems most residential ISPs are not doing IPv6 “the right way.” Or, the designers of IPv6 were a bit more… ~~delusional~~ hopeful let’s say that perfect end-to-end connectivity could be a reality.

What they didn’t account for was that if there was any way ISPs could screw up that dream, then they damn well would do it. And they did.

One of the shortcomings of IPv6 in this scenario is you would normally get a delegated prefix for your internet connection, which you would then divide into other /64 prefixes for your own use, e.g. for VLANs. However, if your ISP changes your prefix (which, at least here in the U.S., is pretty common for residential connections) then it could take some fiddling to get your client machines to notice and then request a new IPv6 address, because the one they were just using is no longer in the network you’ve been assigned. Normally this is transparent when you’re using SLAAC, but if your devices need to do other special things (like have a host firewall) then your firewall rules need to be changed as well. Additionally, using only SLAAC forfeits some of the benefits of using a DHCPv6 server such as being able to control static addressing.

All of this seems to point to the feeling that IPv6 was never designed to handle the bullshittery that ISPs dish out on a regular basis (is a static IPv6 prefix so hard? We only have like… [checks notes] 18,446,744,073,709,551,616 of them).

For me, this manifests as several known problems that I’m too lazy to fix, but recently I finally had enough of one and decided to do something about it.

On my home network, I have (among others) two separate VLANs for my “services” network and my “users” network. The former is for my home lab and other servers which I may or may not expose to the internet via port forwarding, and the latter is where my desktop, laptop, and Android phone reside. Both of these networks are using dual-stack networking.

I’m making great use of NetworkPolicy objects in my Kubernetes cluster for security reasons, but unfortunately I would need to manually change them to accomodate source traffic from the user network in order for it to be allowed.

To fix this, I wrote a Kubernetes operator of sorts that runs as a DaemonSet on each of the nodes. It checks and monitors a config object that looks like so:

apiVersion: apps.fuwafuwatime.moe/v1
kind: IP6UpdateConfig
metadata:
 name: ip6-update-config
spec:
 delegatedPrefixLength: 60
 ciliumCIDRGroups:
 - ciliumCIDRGroupName: ip6-local-lan
 prefixIds:
 - 0
 - 1
 - 2

Then, it determines what node it is running on and checks the corresponding config for that node:

apiVersion: apps.fuwafuwatime.moe/v1
kind: IP6UpdateNodeConfig
metadata:
 name: ip6-update-ram
spec:
 nodeSelector:
 kubernetes.io/hostname: ram.fuwafuwatime.moe
 interface: br0

Next, it checks the IPv6 address on the interface specified in that configs’s spec.interface and performs the bitmasking to calculate the prefixes that would be created from that prefix according to the main config’s spec.delegatedPrefixLength and the listed spec.ciliumCIDRGroups[].prefixIds. Finally, if they differ from what is listed in the named CiliumCIDRGroup, it will update it accordingly with the new CIDRs and post a status to the config object:

Status:
 Conditions:
 Last Transition Time: 2025-04-25T22:18:06Z
 Message: Updated IPv6 address in ip6-local-lan from {} to {"2345:0425:2ca1:1::/64", "2345:0425:2ca1:2::/64", "2345:0425:2ca1:0::/64"}
 Reason: UpdateSucceeded
 Status: True
 Type: Ready

Please note that the above example uses fictitious IPv6 addresses for demonstration.

Then, when the CiliumCIDRGroup is patched, I can just write a corresponding CiliumNetworkPolicy referencing it. Then, whenever my prefix changes, the firewall rules update automatically.

The operator is written in Rust and the full source code can be found here: https://github.com/0xC0ncord/k6u. Please feel free to criticize my Rust skills (but be nice!) — I’m still learning after all.

I have some other ideas for a few features I’d like to add such as only adding CIDRs for specific IPv6 tokens rather than an entire /64, but what I have so far is working well enough for me for the moment. It at least allows me to not have to resort to IPv4-only or using static addressing for accessing my self-hosted email when I’m at home anymore.

’til next time. See ya!

A Small Follow-up to UEFI Secure Boot

Mon, 08 Aug 2022 20:11:51 -0400

This is a follow-up post to UEFI Secure Boot the Right Way in which I answer some questions I have received and to address some confusion and clarify some erroneous information. Not only that, but I will share some additional information that I found out about after publication of that post! 😃

PCR 7 and Microsoft’s Third-Party/Vendor CA Certificate

In my original post I expressed concern that including Microsoft’s third-party CA could potentially downgrade security of the boot process. It turns out that this is partially due to a misunderstanding on how PCR 7 is populated and what actually happens when you bind some value to PCRs in the TPM.

As the system boots, the hashes of all executed code is measured into PCR 7. That is, hashes of each payload are subsequently “appended” to it. The result is that if at any point one of these payloads differs from a previous boot, (as a result of a bootloader update, for example), then PCR 7 will differ between those boots. When you bind a LUKS key to PCR 7, it is guaranteed that the TPM will only reveal the key if the boot process is the same as it was when it was sealed in the first place.

So even if you leave Microsoft’s third-party CA certificate in the trusted store, changing what gets executed during boot will change the value of PCR 7 and the TPM will refuse to reveal the key that was sealed. In short, my original concerns about this are more or less moot.

Microsoft’s Third-Party/Vendor CA Certificate May Be Required on Some Hardware

This is an important piece of information I inadvertently left out. During the hardware initialization process of booting, some devices will have option ROMs with their own code that may need to be executed in order to initialize those devices. The best example I can think of is NVIDIA graphics cards. More often than not, if you install an NVIDIA GPU into your system and enable secure boot, but neglect to install the Microsoft third-party/vendor CA certificate into the trusted store, then the system will halt when booting because the option ROM is not trusted by the firmware.

I actually ran into this exact problem a few years ago with exactly the same setup. I ended up needing to remove the GPU, disabling secure boot, then reinstalling it and ensuring to install Microsoft’s third-party CA certificate before enabling it again.

… Okay, I lied. I ended up RMAing the motherboard thinking I bricked it somehow, and the vendor sent it back, unable to reproduce the problem. Only after did I receive it did I follow the above procedure to fix the issue…

Can malware touch secure boot properties while the OS is running?

Sort of. When UEFI passes execution to the running OS, it does so by calling the ExitBootServices()¹ function. Part of this process involves making most EFI variables and functions read-only or inaccessible, including protected properties governing the operation of secure boot.² Additionally, a UEFI-compliant system’s PK (platform key) variable is made read-only as soon as one is set. There are some other caveats around this which are far too technical to describe here (and frankly, it’s a little too much reading even for me), but the gist is that as long as the system firmware has implemented this process correctly, there isn’t a concern. The secure boot variables that remain writable within the running OS have to be authenticated, meaning that the values you put in have to also be signed and trusted by the platform key. The only secure boot variables with this property are the authorized device signature databases. This is what allows things like Windows updates to add or remove trusted signatures from the device as needed.

So basically, as long as you trust (and/or verify!) that your motherboard is properly UEFI-compliant, then you’re fine.

`sbctl`, the Secure Boot Key Manager

Not long after the original post, I was informed about sbctl which does most of the legwork of the script that I had written previously. I highly recommend using it, as it more or less completely automates the process of generating secure boot keys, enrolling keys, and signing arbitrary EFI executables with them. It will also take care of installing the Microsoft third-party CA certificate if needed!

UEFI Secure Boot the Right Way

Mon, 01 Aug 2022 17:18:34 -0400

Security around the Linux boot process is a bit of a touchy subject. In fact, if you dig deep enough you’ll find that there are many ways to boot a Linux system in general. There’s many different bootloaders, for example, each with their own strengths and weaknesses. In addition, depending on the system’s or the user’s requirements, the system may also require booting with an initramfs or initrd. You can even boot a Linux kernel directly by taking advantage of its CONFIG_EFI_STUB option. While looking into the basics of securing booting on my own machines I found that the documentation for it can sometimes be a bit all over the place, and even then there are many guides out there that were sometimes confusing or didn’t give the exact answers to the questions I still had. All in all, I spent about 2 or 3 weeks really digging into it and only afterwards did I finally come up with a solution that works well enough for me and makes booting my various Linux machines reasonably secure.

This post is a long-awaited one. I originally wanted to publish this a few months ago, but systemd 251 was released which added a new feature that I wanted to cover in addition to everything else I had compiled in. For those who have been patiently waiting for me to finally publish this (you know who you are), thank you.

For the sake of brevity (even though I’ve already well-exceeded it), I will avoid going too much into the technical details for some of the aspects and components of the following material. I will also be skipping over some of the alternatives and different methods of setup. The depth that I will be going into is what I feel is relevant to gain a basic understanding of the things that you, as a user, are setting up if you decide to do it. This is in no way intended to be a “quick guide” or “setup for the lazy”, but that may soon follow if the demand is high enough.

The Current Situation

The first problem with securing the Linux boot process is that the diversity of the many Linux distributions and how they configure the base system also brings fragmentation around booting. This could make setting it up for yourself difficult. Worse, most Linux distributions that do not already have UEFI secure boot just don’t support it, and trying to set it up for yourself may result in system instability after system upgrades. Someone who wants to proceed on such a distribution anyway will need to ensure that they perform some extra steps after system updates or else they may be locked out of their system on next boot or not have a bootable system at all.

Most Linux distributions with existing UEFI secure boot support do so by using the GNU GRUB bootloader with a UEFI shim loader. What happens during boot is that the system EFI firmware will execute the shim loader, which then executes GRUB, which in turn executes the Linux kernel. The shim, when executed, will register itself with the system EFI firmware in such a way that when the system firmware wants to verify a signature for a to be executed EFI executable, it can ask the shim to do so in addition to the firmware’s verification. The result is a sort of parallel signature verification process that allows EFI executables to have their signatures verified by both the system EFI firmware and the shim. The primary advantage of this is that the distribution can sign their own kernels using keys included in the shim and those signatures will be verified without the system owner needing to load those keys into the system firmware.

There are two problems that I (personally) have with this implementation. First, the shim itself is signed by the Microsoft third-party key¹. The corresponding public key is included by default in almost all system EFI firmwares with secure boot support, and Microsoft uses this key pair to sign other objects. What this means is that if an attacker had another object signed with the same key pair, they could boot it on a machine with the key loaded in the system firmware’s trusted keys database. My understanding is that this process is supposed to be restricted to trusted hardware vendors, but it doesn’t seem difficult to maneuver Microsoft into signing something that could potentially have bugs, vulnerabilities, or even backdoors of its own to compromise other machines.

The second problem is that a vulnerability in the GRUB bootloader could potentially allow an attacker to bypass secure boot verification and boot a malicious Linux kernel anyway. GRUB is no stranger to major security bugs that could compromise secure booting, but this is unfair as a simple count of bugs isn’t the best indicator of a project’s security track record. What is noteworthy though, is that vulnerabilities found in GRUB are generally fixed in bulk. This is because every time a massively critical vulnerability is found, the Linux distributions need to get their bootloaders signed by Microsoft again which isn’t a very timely process. Because of this, many small GRUB vulnerabilities will go unfixed for months until GRUB does its yearly major release. Just this past June, a major release of GRUB was announced which fixed multiple known security issues that have been festering for over a year²!

GRUB aside, another major hurdle of Linux boot security is the initrd or initramfs if one is in use. At boot, the kernel will also unpack an initrd into memory which contains tools and kernel modules needed to finish booting the system. For instance, if the root filesystem resides on a filesystem like ZFS which is not available as a built in filesystem driver, the kernel needs to load this kernel module from the initrd at boot in order to mount it. Another case where this is needed is if the machine is (and it should be!) using disk encryption. The kernel alone does not know how to decrypt the disk, let alone ask the user for a passphrase. The initrd will contain the scripts and tools needed to get this going. The initrd is undoubtedly a security-sensitive object. A compromised initrd can potentially contain malicious code that can then infect the rest of the system once it is booted. Unfortunately, current boot security mechanisms like the aforementioned EFI shim loader in combination with GRUB do not in any way protect the initrd from being tampered with because there is no signature verification done against the initrd.

At this point I think it goes without saying that for proper boot security you really really need to have an encrypted disk. An unencrypted and unverified root filesystem really just means that an attacker with physical access can just modify system files in order to cause the system to load malicious code. I didn’t feel it necessary to mention it until now because I figured having an encrypted disk is basically standard security practice by this point and it’s not really related specifically to UEFI secure boot. But anyway, back to secure boot…

The Solution

Ultimately what needs to be done is to somehow protect every sensitive component needed during boot from tampering. The way forward to solve all of these problems is to take advantage of systemd’s boot stub. No, not systemd-boot. The boot stub.

The systemd boot stub is a small EFI executable that does nothing by itself. But, if you use objcopy to build a concatenated blob containing the boot stub, a Linux kernel which has CONFIG_EFI_STUB enabled, the initrd, the kernel’s commandline, and a splash image, you get a single file containing all these components that can be signed for UEFI secure boot and executed. This resulting file is called a unified kernel image. It’s important to note that the systemd boot stub does not actually require systemd on the system to be used. One can easily compile the boot stub separately from the rest of systemd and copy it over to various systems to be used. It is not even necessary to use systemd-boot as the bootloader, as the resulting file with all the bits embedded inside can be used as a standalone EFI payload that can be booted directly from the system EFI firmware.

What we can do is generate our own signing keys for UEFI secure boot and load them into the system firmware’s trusted signature database. Then, we can use the systemd boot stub to create our own unified kernel image and sign them using our generated keys. Finally, we install our unified kernel image to the /boot partition and either add it as an EFI payload in the system firmware or install systemd-boot boot it.

Let’s take it one step at a time, starting with…

Generating Secure Boot Signing Keys

Ideally you’d do this on a separate system that is disconnected from the network and do all of the signing on the same machine as well. That way an attacker who gains code or command execution on the main machine cannot simply copy the UEFI secure boot signing keys because they would not exist on the machine. However, if you are someone like me who tends to update their kernel or initrd fairly frequently this can be impractical. This would mean needing to update the kernel and initrd, then copying them to the machine for signing, sign them, then copying them back. There is definitely some security gains here to do it this way, but I couldn’t be bothered personally. From here on I will assume you are working on the same machine to do the key generation and signing, but if you would like to go the extra mile, simply do these steps on your airgapped machine and follow the procedure I described above every time you need to re-sign your kernel and things.

First, there’s a few packages we will need to install. These packages contain the tools needed to later sign our kernel images. On Gentoo, we can install these by running:

# emerge -av efitools sbsigntools

Then, let’s create a directory on our system where we will store our keys.

# mkdir /etc/secureboot
# chmod 0700 /etc/secureboot

We also set the directory’s permissions to 0700 so that only the root user can view its contents.

Next, let’s create the keys. There are several key pairs that need to be created and loaded into the EFI firmware and they are all related.

First, we will enter the directory where we will store the keys and generate a globally unique identifier (GUID) for the machine:

# cd /etc/secureboot
# uuidgen --random >guid.txt

Then, we will create the platform key:

# openssl req -newkey rsa:4096 -nodes -keyout PK.key -new -x509 -sha256 -days 3650 -subj "/CN=Platform Key/" -out PK.crt
# openssl x509 -outform DER -in PK.crt -out PK.cer
# cert-to-efi-sig-list -g "$(<guid.txt)" PK.crt PK.esl
# sign-efi-sig-list -g "$(<guid.txt)" -k PK.key -c PK.crt PK PK.esl PK.auth
# sign-efi-sig-list -g "$(<guid.txt)" -c PK.crt -k PK.key PK /dev/null rm_PK.auth

The argument given to the /CN= field denotes the canonical name that will be filled into the generated certificate. You can put pretty much whatever you want in this field like the hostname of the machine you are generating keys for, but I like to keep things simple.

The platform key is stored in the UEFI PK variable and it controls access to that variable as well as the KEK variable, which is where the key exchange key will be stored.

The above commands will have also created a file named rm_PK.auth. This file can be installed to the PK variable in order to remove the platform key in the event that it is needed.

Next, we need to create the key exchange key.

# openssl req -newkey rsa:4096 -nodes -keyout KEK.key -new -x509 -sha256 -days 3650 -subj "/CN=Key Exchange Key/" -out KEK.crt
# openssl x509 -outform DER -in KEK.crt -out KEK.cer
# cert-to-efi-sig-list -g "$(<guid.txt)" KEK.crt KEK.esl
# sign-efi-sig-list -g "$(<guid.txt)" -k PK.key -c PK.crt KEK KEK.esl KEK.auth

The key exchange key is used to update the signature database or to sign binaries that will be trusted by the system firmware for execution.

Finally, we will create the signature database.

# openssl req -newkey rsa:4096 -nodes -keyout db.key -new -x509 -sha256 -days 3650 -subj "/CN=Signature Database Key/" -out db.crt
# openssl x509 -outform DER -in db.crt -out db.cer
# cert-to-efi-sig-list -g "$(<guid.txt)" db.crt db.esl
# sign-efi-sig-list -g "$(<guid.txt)" -k KEK.key -c KEK.crt db db.esl db.auth

The signature database is stored in the system’s db variable and is what is used to validate signed EFI binaries and other payloads when secure boot is enabled.

Installing Keys and Signing Things

Now that our keys are created, we can start signing objects with them. The way to do this is via the sbsign utility:

# sbsign \
 --key /etc/secureboot/db.key \
 --cert /etc/secureboot/db.crt \
 --output /boot/EFI/systemd/systemd-bootx64.efi \
 /boot/EFI/systemd/systemd-bootx64.efi

The above command will sign the systemd-boot bootloader and overwrite the original file. If you wish to preserve the original, unsigned file, simply point the --output argument to a different location.

In order to actually take advantage of signature verification, though, we need to install our keys into the EFI firmware. We can do this from the running operating system using the efi-updatevar utility:

# efi-updatevar -f /etc/secureboot/PK.auth PK
# efi-updatevar -f /etc/secureboot/KEK.auth KEK
# efi-updatevar -f /etc/secureboot/db.auth db

If this is not an option, or you’d like to do this manually, you can instead copy the *.auth files to the /boot directory or some other device that we can read from the system firmware (usually an external FAT partition). Then, from within the EFI firmware settings we can manually load the keys. Finally, we can enable secure boot from within the firmware.

Warning

If you are using a bootloader such as systemd-boot, you will need to ensure to sign the bootloader itself or your system won’t boot!

Either way you do this, you should ensure that you protect the firmware settings from modification. On most motherboards this is generally just an administrator password and admittedly could be better, but unfortunately this is what we are stuck with. The reason you should do this is that an attacker with physical access could simply wipe the secure boot keys or simply turn secure boot off in order to trick the system into booting into an insecure state.

Putting It All Together

Before any of what we are about to do can work, your kernel must have CONFIG_EFI_STUB enabled. This is needed to allow the system firmware (or systemd’s boot stub) to pass execution directly to the kernel.

To ensure that the initrd is protected from tampering as well as the kernel image, we’ll need to combine the initrd and kernel into a single object for signing. As mentioned earlier, the systemd boot stub is a small EFI executable that can be combined with a Linux kernel, an initrd, the kernel commandline, and a splash image in order to generate a single file that can be signed for secure boot. For this we use objcopy to combine the required components:

# objcopy --add-section .osrel="/etc/os-release" \
 --change-section-vma .osrel=0x20000 \
 --add-section .cmdline="/etc/kernel/cmdline" \
 --change-section-vma .cmdline=0x30000 \
 --add-section .splash="/etc/kernel/splash.bmp" \
 --change-section-vma .splash=0x40000 \
 --add-section .linux="/boot/vmlinuz-5.15.58-gentoo" \
 --change-section-vma .linux=0x2000000 \
 --add-section .initrd="/boot/initramfs-5.15.58-gentoo" \
 --change-section-vma .initrd=0x4000000 \
 "/usr/lib/systemd/boot/efi/linuxx64.efi.stub" \
 "/boot/EFI/Linux/linux-5.15.58-gentoo.efi"

There’s a lot to unpack here in this command, but let’s go through it piece by piece:

We embed various components into the resulting file. These are:

/etc/os-release - the OS release information file
/etc/kernel/cmdline - a plain text file containing the kernel commandline
/etc/kernel/splash.bmp - a splash image in BMP format
/boot/vmlinuz-5.15.58-gentoo - the kernel image
/boot/initramfs-5.5.15-gentoo - the initrd

It should be noted that on Gentoo, the VERSION_ID field in /etc/os-release may not exist. This field is read by systemd-boot when populating the list of valid bootable entries. This is not needed if you intend to boot the image directly from firmware, but if you are using systemd-boot you will need this field. I have mine set to "2.8".

Each argument which adds a file is separated by an argument which sets the offset for the file to be inserted. I suggest not to change these as I’m pretty sure they may be hard-coded but don’t quote me on that.

Finally, near the end of the command we see the path to the systemd boot stub and the output path to the resulting file.

For convenience, what I have done on my systems is install the systemd-boot installkernel script (installkernel-systemd-boot on Gentoo) along with a kernel install script hook I wrote that automates this process whenever I rebuild the kernel:

#!/usr/bin/env bash

set -Eeuo pipefail

COMMAND="${1}"
KERNEL_VERSION="${2}"
BOOT_DIR_ABS="${3}"
KERNEL_IMAGE="${4-}"

# If KERNEL_INSTALL_MACHINE_ID is defined but empty, BOOT_DIR_ABS is a fake directory.
# So, let's skip creating a unified kernel image.
if ! [[ ${KERNEL_INSTALL_MACHINE_ID-x} ]]; then
 exit 0
fi

if [[ -d "${BOOT_DIR_ABS}" ]]; then
 INITRD="initrd"
 TARGET="../../EFI/Linux/linux-${KERNEL_VERSION}.efi"
else
 BOOT_DIR_ABS="/boot"
 INITRD="initramfs-${KERNEL_VERSION}.img"
 TARGET="linux-${KERNEL_VERSION}.efi"
fi

die() {
 echo "${1}"
 exit 1
}

case "${COMMAND}" in
 add)
 [[ -f "${KERNEL_INSTALL_BOOT_ROOT}"/../etc/secureboot/db.key && -f "${KERNEL_INSTALL_BOOT_ROOT}"/../etc/secureboot/db.cer ]] || die "No valid keys found in /etc/secureboot!"
 [[ -f "${KERNEL_INSTALL_BOOT_ROOT}"/../etc/kernel/cmdline ]] || die "Cannot proceed: No such file '/etc/kernel/cmdline'"
 [[ -f "${KERNEL_INSTALL_BOOT_ROOT}"/../etc/os-release ]] || die "Cannot proceed: No such file '/etc/kernel/splash.bmp'"

 args=(
 --add-section .osrel="${KERNEL_INSTALL_BOOT_ROOT}/../etc/os-release"
 --change-section-vma .osrel=0x20000
 --add-section .cmdline="${KERNEL_INSTALL_BOOT_ROOT}/../etc/kernel/cmdline"
 --change-section-vma .cmdline=0x30000
 --add-section .splash="${KERNEL_INSTALL_BOOT_ROOT}/../etc/kernel/splash.bmp"
 --change-section-vma .splash=0x40000
 --add-section .linux="${BOOT_DIR_ABS}/linux"
 --change-section-vma .linux=0x2000000
 )

 if [[ -f "${BOOT_DIR_ABS}/${INITRD}" ]]; then
 echo "Using initrd ${INITRD}"
 args+=(
 --add-section .initrd="${BOOT_DIR_ABS}/${INITRD}"
 --change-section-vma .initrd=0x4000000
 )
 fi
 args+=(
 "${KERNEL_INSTALL_BOOT_ROOT}/../usr/lib/systemd/boot/efi/linuxx64.efi.stub"
 "${BOOT_DIR_ABS}/${TARGET}"
 )

 echo "Creating unified kernel image..."
 objcopy ${args[@]}

 echo "Signing the unified kernel image..."
 sbsign \
 --key "${KERNEL_INSTALL_BOOT_ROOT}"/../etc/secureboot/db.key \
 --cert "${KERNEL_INSTALL_BOOT_ROOT}"/../etc/secureboot/db.crt \
 --output "${BOOT_DIR_ABS}/${TARGET}" \
 "${BOOT_DIR_ABS}/${TARGET}"

 echo "Removing old loader entry..."
 rm -vf "${BOOT_DIR_ABS}/../../loader/entries/${KERNEL_INSTALL_MACHINE_ID}-${KERNEL_VERSION}.conf"
 ;;
 remove)
 rm -f -- "${BOOT_DIR_ABS}/${TARGET}"
 ;;
esac

The above script, when installed with executable permissions to /etc/kernel/install.d/99-secureboot.install, take care of combining all these files together and installing them to the proper location.

Going a Step Further

systemd-cryptenroll is a utility that enrolls new key slots into an existing LUKS device. These new key slots can be handled by the systemd-cryptsetup tool such that the user can be prompted to decrypt the device. The advantage of these two tools in tandem is that we can enroll key slots that require the user to do more than just enter a passphrase to unlock the device. systemd-cryptenroll has support for PKCS#11-compliant security tokens, FIDO2 tokens, and TPM2 devices. The latter is what we will be focusing on here.

At a high level we can take advantage of a TPM2 device on the system to seal the key to decrypt the root filesystem at boot, then have systemd-cryptsetup ask us for a passphrase in order to ask the TPM to unseal the key. Why ask the TPM to unseal the key? Well, this is because when we seal a key or some other secret data in the TPM, the TPM will only unseal it if the platform configuration registers (PCRs) match. What happens during boot is the TPM will have stored hashes of each of the different components that were used during bootup, and if we seal a secret in the TPM against these hashes, the TPM will only unseal the secret if they match. What this means for us is that we can seal the disk decryption key in the TPM against one or more TPM2 PCRs, and the system has to be in a trusted state during boot in order for the disk to be decrypted. For me this means that the TPM will only hand over the disk decryption key if secure boot is enabled and all the signatures of the different payloads during boot were successfully verified.

systemd-cryptenroll allows us to either add a key slot that requires a passphrase or add a key slot that does not. The latter will allow the system to boot without a passphrase so long the PCRs match. This is convenient in that we can be sure that the system is in a secure state when it boots, but the risk is that if the machine does something after booting is finished that could potentially be compromised or used to leverage access, an attacker can do so. A prime example of this is this lovely penetration test conducted by Dolos Group³. What they found was that even though the machine they were trying to use for attack was properly configured for secure boot, the machine would automatically connect to the company’s internal network via an automatic VPN connection that was established after boot. They were able to leverage this in order to gain a foothold in the internal network even though the machine itself was sufficiently secured. The solution? Require a PIN or passphrase to continue booting even though the device was booted in a secure state. The takeaway is that any potential attack surface that exists on the machine after boot cannot be used if the attacker does not know the PIN or passphrase to complete the boot process.

To ensure that systemd-cryptenroll can detect the TPM2 device, we can use:

# systemd-cryptenroll --tpm2-device=list
PATH DEVICE DRIVER
/dev/tpmrm0 IFX0785:00 tpm_tis

This will print the path to the TPM device in /dev as well as information about it if it is found. If you do not see a device here, make sure you actually have a TPM2 device installed and that the kernel can load the appropriate driver for it.

Since we have only one TPM2 device on this system, it is safe to use --tpm2-device=auto when issuing commands that will work on the TPM. Otherwise, we would need to specify the path to the TPM2 device in this argument. To enroll a new key slot using TPM2 PCR 7, which is the PCR that stores the secure boot state, we can use:

# systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs=7 /dev/nvme0n1p2

This command will ask for the current passphrase to unlock the /dev/nvme0n1p2 LUKS partition and then enroll a new key slot against PCR 7 into it. At boot, this partition will automatically be decrypted at boot as long as PCR 7 matches. If, instead, we want to also enter a PIN or passphrase to unlock it in addition to being in a secure boot state, we can use:

# systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs=7 --tpm2-with-pin=true /dev/nvme0n1p2

This command is the same as the above except we add --tpm2-with-pin=true which tells systemd-cryptenroll to enroll a key slot that requires both PCR 7 hashes to match as well as for the user to enter a PIN or passphrase. When we enter this command, it will again ask us to enter the current passphrase to unlock /dev/nvme0n1p2 and then ask for a PIN to use for unlocking via the TPM. Although it says PIN for this prompt (and even when unlocking), non-numeric characters are allowed.

We can also use other PCRs available in the TPM other than just PCR 7. The systemd-cryptenroll man page⁴ has a table of well-known PCRs and their contents. For example, PCR 0 will contain a measurement of the core system firmware executable code. So, if you wanted to ensure that the system firmware itself was not updated or modified since enrolling, you can include it:

# systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs=0,7 --tpm2-with-pin=true /dev/nvme0n1p2

This gives an extra layer of defense in verifying that not only secure boot is in a known good state, but also checking other components. Keep in mind that by enabling the use of other PCRs, updating any of these components (even legitimately) will stop the TPM from unsealing secrets sealed against these PCRs. Windows’s BitLocker for example only uses PCR 7⁵, specifically to avoid boot issues when the system owner updates the system firmware.

At this point the next best thing to do would be to enroll a recovery key to the partition using the --recovery-key argument to systemd-cryptenroll and then securely storing it somewhere and wiping the original passphrase slot. This would ensure that the LUKS partition can only be unlocked by using a PIN/passphrase in conjunction with being in a secure boot state. The downside of this is that if something were to ever go wrong with the boot process, you would likely be locked out of the system until you have access to the recovery key.

"Jammy Jellyfish" jammed my girlfriend's laptop

Thu, 28 Apr 2022 10:20:40 -0400

My girlfriend has been using Kubuntu 20.04 on her laptop almost since it was released. Her intent was to become more familiar with Linux and computing in general, which I am more than happy to help with. She was very enthusiastic from the beginning about delving into the innards of a Linux desktop, but I didn’t want to overwhelm her with something as technical as Gentoo just yet. So, because of some friendly recommendations and from my own personal experience I settled on helping her get set up with Kubuntu 20.04 Focal Fossa.

It has been almost two years since we initially got her going. She spent a lot of time customizing her KDE desktop just the way she liked, with a set of gaudy icons, wallpapers, and animations that honestly were a bit out of my style. But, nonetheless, she was happy with it and so was I. For the purposes of this post I will forego mentioning how often I needed to remind her to update her system and get to the point.

About a week ago I reminded her to run the regular update mechanism via apt on her machine, only this time I suggested we upgrade the machine to Kubuntu 22.04 which was just freshly released. I had only heard of some minor complaints regarding the release, mostly around the “forced” packaging of Firefox as a snap and so I decided it was a good idea for her to do the upgrade, especially since I was physically present. After running the appropriate apt commands to update the local repo metadata and upgrade the installed packages, we performed the dist-upgrade command and let it run. After about 20 minutes of churning and answering some questions during the upgrade, the machine was ready to reboot. And reboot it did.

Immediately we were presented with a problem: her laptop was hanging at the boot process. Not long after exiting the initramfs, several systemd services would fail to start including some critical ones like systemd-logind. To cut a long story short I spent the next hour or so troubleshooting the machine with a LiveUSB I was carrying around but without success. It was getting late at this point and so I went to bed with the intent of continuing in the morning.

When I woke up, I got right to work with seeing if I could recover the machine. I tried to see what (if any) problematic service was hanging boot, disabling potentially unneeded services, and rebuilding the initramfs and Grub config, all without success. I wasn’t able to spend more than maybe an hour in the morning before leaving for work unfortunately. When I got home I went right back to trying to fix the machine before we decided it was best to just do a clean reinstall. For some reason, however, the installer in Kubuntu would hang too! I was able to navigate through some of the menus up until I was supposed to select the disk partitioning scheme, but the menu would never finish loading. I tried this several times before just giving up.

In the end I decided to just install Gentoo on the laptop. Unfortunately for my girlfriend, she will need to deal with potentially long compile times and a much steeper learning curve than we were hoping. Lucky for me though, installing Gentoo on the machine makes me feel much more comfortable helping her with technical issues and generally just feeling at home on her machine as if it were my own. Plus, this alleviates the need to install it later if she ever decides to take the next step in learning what I feel is a true Linux system.

I don’t have any hard feelings towards Kubuntu or Ubuntu at this point, but I have to admit that this whole ordeal did leave me and my girlfriend quite a bit frustrated. I do get the feeling that we did something wrong during the upgrade, but unfortunately we are beyond the point of finding out what really happened that led to her laptop being in a broken state. Besides, in the long run I think that shooting for the moon and installing Gentoo on the machine means my girlfriend will be able to get a lot closer to the machine and learn the innards a lot sooner than anticipated. Perhaps I’m getting ahead of myself though, since she will need to get in the habit of doing routine updates first. 😜

’til next time. See ya!

How not carefully reading the docs cost me a heart attack and a change of pants

Tue, 26 Apr 2022 13:29:52 -0400

A couple weeks ago I was coming into work and wanted to quickly finish up some experimentation I was doing with regards to setting up automatic disk decryption at boot using TPM 2.0 and LUKS. I have been using full disk encryption (FDE) for a while now but I wanted to try out the different ways that systemd-cryptsetup supports unlocking the disk. Of the various methods, it seems that using a TPM 2.0 device to unlock the disk, given the validity of the system firmware and secure boot state, was most interesting. The most desireable approach is actually to use TPM 2.0 along with a passphrase or PIN, but this functionality is not yet available as it will be released in systemd v251.

Anyways, I had previously added a slot to the TPM consisting of PCRs 0,2,4,7 and found that the disk was not automatically unlocking. I could verify that I had the slot added with:

# systemd-cryptenroll /dev/nvme0n1p2

I wanted to remove the old slot and try again, so I issued this command:

# systemd-cryptenroll /dev/nvme0n1p2 --wipe-slot=0,2,4,7

Then, I went to add a new slot but with different PCRs, so I did:

# systemd-cryptenroll /dev/nvme0n1p2 --tpm2-device=auto --tpm2-pcrs=0,7

This command prompted me for a passphrase like usual, but for some reason it wouldn’t accept my passphrase. When I reissued a command to view the slots, this is what I saw:

# systemd-cryptenroll /dev/nvme0n1p2
SLOT TYPE
 1 tpm2

My passphrase was gone! 😱

At this point I was in a bit of a panic. I tried a few more times out of insanity to try and add in the passphrase again so that I could work on my encrypted disk, but to no avail. I started to collect myself and had a couple realizations. First, I couldn’t think of any way to restore the passphrase into the LUKS header of the disk without just reformatting it, and second, my disk was still decrypted at this point in time.

So, while continuously being paranoid of a kernel panic or largely improbable cosmic event knocking out my laptop and costing all my data, I continued to work until lunch came around, at which point I went back to the house and grabbed my SATA to USB hard drive adapter and a spare 8TB hard disk drive I had laying around. (Okay, the drive was meant to go into my desktop as a replacement but I have been too lazy to do it as of late.)

I brought the two devices back to the office and continued to work until the closing of the day. I immediately hooked up the drive to my laptop and began copying over all my data with an rsync command. I left my laptop at the office that day (locked to the desk, mind you) while the backup operation ran overnight.

The next morning I came in to see that it had thankfully finished. With my fingers crossed, I rebooted the laptop back into a Gentoo LiveDVD environment and reformatted the LUKS partition with my old passphrase. Then, I began recreating my LVM volumes on top of it and adjusting my /etc/fstab accordingly. After the volumes were all mounted, I ran another rsync to begin the restoration process and found something else to do at work while this was running. Needless to say this also took basically the entire day, and I left my laptop at the office for a second night while this completed.

At the dawn of the final day, I returned to my laptop to see it had finished. After verifying that my /etc/fstab was correct and making the same adjustments to the kernel commandline, I rebooted the machine and was greeted with a pleasant TTY asking for my credentials. 👏

The last thing I needed to do after rebooting was fix the SELinux contexts on the machine with a simple rlpkg -ar and manually fix the contexts on the various mountpoints for the root inode of the volumes (sit tight for a minor writeup on this as well).

After all was done it was like I had never screwed up in the first place. What I didn’t mention was that I was also researching on the side what exactly I did wrong while the recovery was taking place. You may have already guessed it, but my error was in thinking that the --wipe-slot argument took PCRs to wipe and not the slot itself. Normally when you encrypt a disk with LUKS and a passphrase, the passphrase goes into slot 0. When issuing the above wipe command, I ended up removing the passphrase slot entirely. If the TPM unlocking had worked, I could in theory have not worried so much about it, however I would not be able to update the kernel or initramfs without breaking the disk decryption process. Additionally, I never got it to work in the first place with those PCRs.

I am writing this from the same laptop a couple weeks since this happened with a fully working disk decryption with TPM 2.0 setup. It turns out that in addition to the errors I made above, dracut currently has a bug that omits some of the required libraries for systemd-cryptsetup to do its job. I had to manually include these in my dracut config for the time being until a new version with the fix is released. Lastly, I opted to stick with PCRs 0,7 for the time being as this allows me to update my kernel and initramfs together without needing to add a new slot for them every time.

Going forward, I intend to add a recovery key to the disk in the event that something like this happens again. That way, I should still be able to manipulate the LUKS header in case I do something similarly silly in my experimentation.

Hopefully this was an interesting read and a lesson to carefully read the corresponding documentation when working on particularly sensitive projects.

’til next time, see ya! 👋

License and Copyright

Mon, 04 Oct 2021 13:57:30 -0400

Some icons are provided by Font Awesome under a CC BY 4.0 License.

The Terminus font is licensed under the GPL-2 and the OFL-1.1.

The OpenSans font is licensed under the Apache-2.0 license.

The hugo-notice component is licensed under the GPL-3.0 or later.

All other content is licensed under a CC BY 4.0 License, unless otherwise explicitly noted.

Gentoo Clang Toolchain Initial Setup

Sun, 03 Oct 2021 14:31:34 -0400

Clang is pivoted to become the next-generation C/C++ compiler for modern systems. It aims to provide better diagnostics, be easier to integrate with IDEs, and have a license that is more compatible with commercial products.¹ On Gentoo, Clang is already available as a compiler that can be used system-wide instead of GCC. The primary motivation for this is to take better advantage of link-time-optimization (LTO)² features. While GCC does support LTO, it only supports “full” LTO, which may not be feasible on lower-end systems. This is because full LTO requires merging all of the program’s input into a single module at once, which can be very demanding on systems with low memory or compute power. Clang, on the other hand, supports “ThinLTO,”³ which aims to be a viable alternative to full LTO by only merging summaries of LTO bitcode at link time. This allows for fast and efficient optimizations to be made to the program without requiring the amount of resources that full LTO otherwise would.

The other primary motivation for implementing this on Gentoo is to build programs with Control Flow Integrity (CFI)⁴ enabled. CFI is a series of compile-time hardening features that can be utilized to mitigate or eliminate various code-reuse attacks, among other things. Clang employs a variety of different CFI schemes, none of which are currently (nor are there any plans to be) supported by GCC. We can take advantage of Clang’s CFI to further harden the system by building our programs with it, and therefore reduce attack surface and prevent potentially malicious behavior as a result of a compromise. Lastly, CFI is available⁵ as part of the mainline Linux kernel, although at the time of writing, it is only currently supported on ARM64 platforms. Support for x86_64 is in the works⁶, and will likely land in upstream Linux in 5.16.

Getting started with using Clang as a system-wide compiler isn’t too difficult - it’s mostly supported in Gentoo⁷, with only a few various edge cases floating around in my personal experience.

Warning

While I use Clang as a system-wide compiler on Gentoo systems in production, it is not without its problems. First and foremost, GCC is the defacto standard compiler on Gentoo. While Clang support does exist, it is not a high priority and as such I do not recommend removing GCC as an alternative compiler. This is because many packages still refuse to compile (or run after being built) with Clang, although that number is shrinking.

Additionally, for those problematic packages that do not build cleanly, there are a series of workarounds that you must employ in order to get them to build, all of which I will detail in this guide. I do not, however, consider any of these practices recommended for use by an inexperienced Gentoo user who is afraid of potentially breaking their system.

To get started, we need to initially build the entire Clang/LLVM toolchain using GCC. I find it is much easier to create a package set so that we can emerge the set as opposed to each package individually.

# /etc/portage/sets/llvm-toolchain
sys-devel/llvm[abi_x86_32,gold]
sys-devel/clang[abi_x86_32,default-compiler-rt,default-libcxx,default-lld,static-analyzer]
sys-devel/lld
sys-devel/clang-runtime[abi_x86_32,compiler-rt,libcxx,sanitize]
sys-libs/compiler-rt[clang]
sys-libs/compiler-rt-sanitizers[clang,asan,cfi,dfsan,gwp-asan,hwasan,libfuzzer,lsan,memprof,msan,profile,safestack,scudo,tsan,ubsan,xray]
sys-libs/libcxx[abi_x86_32,libunwind,static-libs]
sys-libs/libcxxabi[abi_x86_32,libunwind,static-libs]
sys-libs/libunwind[abi_x86_32,static-libs]

Note

The text inside square brackets ([]) next to each package name denotes USE flags required in this set. abi_x86_32 is the USE flag to enable 32-bit support if you intend on using Clang to build 32-bit packages. If you do not need this, remove the abi_x86_32 USE flag.

Note

The compiler-rt and compiler-rt-sanitizers packages and USE flags allow Clang to use its own static analyzer when performing compile-time sanitization. The libcxx and libcxxabi packages and USE flags allow Clang to use libc++ as the default C++ standard library instead of GCC’s stdlibc++. We will need these if you want to set up CFI support later, but they are otherwise optional.

It should be noted that switching between libc++ and stdlibc++ should not be taken lightly, as they are not ABI-compatible. However, some packages (namely Chromium) refuse to build with CFI unless its dependencies are built with libc++.

Note

llvm-libunwind is the better option over libunwind. However, there are several packages that still hard-depend on libunwind as opposed to offerring llvm-libunwind as an alternative, namely Samba. If you do not use any of these problematic packages, you are encouraged to use llvm-libunwind instead and enable the corresponding USE flags.

After that, enable the USE flags in this set, either globally or on a per-package basis.

# /etc/portage/make.conf
...
USE="llvm clang gold libcxx libcxxabi
 default-libcxx compiler-rt
 default-compiler-rt default-lld libunwind"
...

Finally, we can emerge the set.

emerge -av @llvm-toolchain

Before we begin using it, the Clang toolchain needs to be bootstrapped, or in other words, rebuilt using Clang itself. We need to set Clang and the LLVM tools as the defaults for the system. Additionally, we should start enabling the relevant CFLAGS and CXXFLAGS. It is imperative that we enable LTO now if we intend to use it, as Clang or LLVM may emit spurious errors when building LTO-enabled packages when they themselves are not built with LTO (also, we get a nice performance boost to compilation).

# /etc/portage/make.conf

# Use clang instead of gcc
CC="clang"
CXX="clang++"
LD="ld.lld"

# Use the proper tools capable of dealing with LLVM bitcode
AR="llvm-ar"
NM="llvm-nm"
RANLIB="llvm-ranlib"
STRIP="llvm-strip"
OBJDUMP="llvm-objdump"
OBJCOPY="llvm-objcopy"
OBJSIZE="llvm-objsize"
STRINGS="llvm-strings"
READELF="llvm-readelf"

# Standard flags that normally don't cause problems
COMMON_FLAGS="-march=native -mtune=native -O3 -pipe -fomit-frame-pointer"

# Flags for compile-time hardening
HARDENED_FLAGS="${COMMON_FLAGS} -fPIE -fstack-protector-strong -D_FORTIFY_SOURCE=2 -fstack-clash-protection"

# LTO flags. We use LTO_FLAGS_THIN here but we declare LTO_FLAGS
# with full LTO as it will be easier to switch to later
LTO_FLAGS="-flto"
LTO_FLAGS_THIN="-flto=thin"

CFLAGS="${HARDENED_FLAGS} ${LTO_FLAGS_THIN}"
# Use libc++ as the standard C++ library
CXXFLAGS="${HARDENED_FLAGS} ${LTO_FLAGS_THIN} -stdlib=libc++"
# Hardened LDFLAGS and use lld as the default linker
LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,-S -Wl,-z,now -Wl,-z,relro -fuse-ld=lld -Wl,-unwindlib=libunwind"

Note

If you do not intend to use libc++ as the standard C++ library, omit the corresponding argument from CXXFLAGS.

After that, rebuild your toolchain.

emerge -av @llvm-toolchain

At this point you should now have a functioning Clang toolchain. But, we need to set up our environment to work around various problematic packages. To do this, we create package env files in /etc/portage/env with various flags that override our defaults in make.conf. Then, we can tell Portage to use these overrides in /etc/portage/package.env.

# /etc/portage/env/compiler-gcc

CC="gcc"
CXX="g++"
AR="ar"
NM="nm"
RANLIB="ranlib"
STRIP="strip"
OBJDUMP="objdump"
OBJCOPY="objcopy"
OBJSIZE="objsize"
STRINGS="strings"
READELF="readelf"
CFLAGS="${HARDENED_FLAGS} ${LTO_FLAGS}"
CXXFLAGS="${HARDENED_FLAGS} ${LTO_FLAGS} -nostdinc++ -I/usr/include/c++/v1"
FCFLAGS="${HARDENED_FLAGS} ${LTO_FLAGS}"
FFLAGS="${HARDENED_FLAGS} ${LTO_FLAGS}"
LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,-S -Wl,-z,now -Wl,-z,relro -fuse-ld=lld -nodefaultlibs -lc++ -lc++abi -lm -lc -lgcc_s -lgcc"

This env file overrides various flags in make.conf when building a package. To apply this to a package, we provide it in an entry in /etc/portage/package.env like so:

# /etc/portage/package.env/podman

app-emulation/podman compiler-gcc

The above env file tells Portage to use GCC as the compiler, but still using lld as the linker and libc++ as the standard C++ library. Not all packages can be built this way, so we simply narrow down the flags closer to the Gentoo defaults until it builds. This is usually a series of trial and error steps, but in my experience I have only found a small handful of packages that can only be built with the defaults.

Tip

You can stack env flags on a package by declaring more than one in /etc/portage/package.env. When this is done, the flags will be applied in order from left to right. You can use this to your advantage by keeping your env files slim and only containing the flags for a specific purpose.

What follows are additional env files I use that can help with this entire procedure.

# /etc/portage/env/linker-gold

# This env tells Portage to use the `gold` linker.
# While `gold` is technically deprecated, it is still
# functional and supports LTO, unlike the `bfd` linker
# which is the Gentoo default.
LD="ld.gold"
LDFLAGS="${LDFLAGS} -fuse-ld=gold"

# /etc/portage/env/no-lto

# This env disables LTO completely for the package.
# Some packages simply refuse to build with LTO.
CFLAGS="${CFLAGS} -fno-lto"
CXXFLAGS="${CXXFLAGS} -fno-lto"

# /etc/portage/env/linker-bfd

# This env tells Portage to use the `bfd` linker.
LD="ld.bfd"
LDFLAGS="${LDFLAGS} -fuse-ld=bfd"

# /etc/portage/env/with-lomp

# This tells Portage to explicitly link the package
# with `libomp`.
#
# For some reason, some packages do not declare all
# required shared objects during the link phase when
# built with Clang, even though they compile fine
# otherwise (Audacity is one of these).
#
# If you get any compilation errors that say things
# like "unresolved symbol", try identifying what
# shared object provides that symbol and link to it
# explicitly like above. Some other libraries I have
# seen affected by this include `libm`, `libz`, and
# `libdl`, but these are usually rare.
CFLAGS="${CFLAGS} -lomp"
CXXFLAGS="${CXXFLAGS} -lomp"

# /etc/portage/env/compiler-gcc-libstdc++

# Use GCC as the compiler with `libstdc++` as the
# standard C++ library. There isn't really an easy
# way to explicitly specify the standard C++ library
# with GCC like you can with Clang's `-stdlib=`
# argument, so unfortuantely we have to declare
# these flags in combination with setting GCC.
CC="gcc"
CXX="g++"
AR="ar"
NM="nm"
RANLIB="ranlib"
STRIP="strip"
OBJDUMP="objdump"
OBJCOPY="objcopy"
OBJSIZE="objsize"
STRINGS="strings"
READELF="readelf"
CFLAGS="${HARDENED_FLAGS} ${LTO_FLAGS}"
CXXFLAGS="${HARDENED_FLAGS} ${LTO_FLAGS}"
LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,-S -Wl,-z,now -Wl,-z,relro -fuse-ld=gold"

Whenever troubleshooting a problematic package, I usually follow this order of flags, adding on additional ones until the package builds.

Disable LTO.
Use GCC as the compiler.
Use gold as the linker.
Use GCC as the compiler and libstdc++ as the standard C++ library.
Use bfd as the linker.

In the next part of this series of guides we will move into building packages with CFI, and then finally I will detail my process for building Ungoogled Chromium with CFI as well.

See you next time!

Nextcloud cron with SELinux

Sat, 24 Oct 2020 13:10:00 -0400

If you run your own Nextcloud instance, one of the things you will need to do when finalizing setup is to configure Nextcloud’s internal cron service.

The default and least reliable option is AJAX, which will run the cron service each time a Nextcloud page is requested. The immediate problem here is that if your Nextcloud instance does not have much continuous usage (like mine), especially from a web browser, the cron service will not run as often as it needs to.

The recommended option is system cron, which requires setting up a system cronjob like so:

*/5 * * * * nginx php -f /var/www/nextcloud/htdocs/cron.php

This cronjob will call the cron.php service directly running under the nginx user. On typical systems this is fine, but if you are using SELinux then you will notice some problems like the following:

node=megumin type=AVC msg=audit(1603560001.338:3684): avc: denied { execmem } for pid=30481 comm="php" scontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=process permissive=1
node=megumin type=AVC msg=audit(1603560001.338:3684): avc: denied { execstack } for pid=30481 comm="php" scontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=process permissive=1

What’s happening here is the cron service is running under the system cron context, which does not have permissions to execute writable memory, which is required for PHP to work.

These are two very sensitive and potentially dangerous permissions, so if you are like me, you may not want to simply grant these permissions to the system cron context. Instead, you think to try and create a stub script in order to set up a process transition like so:

#!/bin/sh
php -f /var/www/nextcloud/htdocs/cron.php

type phpfpm_cron_exec_t;
application_executable_file(phpfpm_cron_exec_t)
cron_system_entry(phpfpm_t, phpfpm_cron_exec_t)

Now we can label our script as phpfpm_cron_exec_t, make it executable, and tell our cronjob to execute this file.

But now we have new problems:

node=megumin type=AVC msg=audit(1603558690.194:3611): avc: denied { map } for pid=27906 comm="sh" path="/bin/bash" dev="sda4" ino=13238332 scontext=system_u:system_r:phpfpm_t:s0 tcontext=system_u:object_r:shell_exec_t:s0 tclass=file permissive=1
node=megumin type=AVC msg=audit(1603558690.194:3611): avc: denied { execute_no_trans } for pid=27906 comm="php-fpm" path="/bin/bash" dev="sda4" ino=13238332 scontext=system_u:system_r:phpfpm_t:s0 tcontext=system_u:object_r:shell_exec_t:s0 tclass=file permissive=1
node=megumin type=AVC msg=audit(1603558690.194:3611): avc: denied { read open } for pid=27906 comm="php-fpm" path="/bin/bash" dev="sda4" ino=13238332 scontext=system_u:system_r:phpfpm_t:s0 tcontext=system_u:object_r:shell_exec_t:s0 tclass=file permissive=1
node=megumin type=AVC msg=audit(1603558690.194:3611): avc: denied { execute } for pid=27906 comm="php-fpm" name="bash" dev="sda4" ino=13238332 scontext=system_u:system_r:phpfpm_t:s0 tcontext=system_u:object_r:shell_exec_t:s0 tclass=file permissive=1

The phpfpm_t type does not have the permissions needed to run a shell and execute the php binary.

So that takes us back to square one. Let’s take another look at our options again. We also have the option of using a Webcron service. It took me a minute to figure this out since I had never heard of it before, but essentially you can use a third-party service which will send HTTP GET requests (or whatever you configure) to endpoints on your webserver, which will trigger PHP to do something on the backend. After realizing what this could do for us, I came up with the following solution.

We can use the Webcron service and simply curl our cron.php internally using the system cron. Easy!

So we create a cronjob like so:

*/5 * * * * root curl https://nextcloud.example.com/cron.php >/dev/null

And then we tell Nextcloud that we are using a Webcron. That’s it!

Additionally, you could even restrict the cron service endpoint to the local host or network only. To do this, you can insert this block into your Nextcloud’s server definition in Nginx:

location = /cron.php {
 allow 127.0.0.1/32;
 deny all;
}

Finally, we have Nextcloud’s cron service set up, fully working, and playing nice with SELinux.

Wrong run_init Authentication

Tue, 22 Sep 2020 15:11:42 -0400

Today I noticed that for some reason on one of my Gentoo boxes, I needed to authenticate as root in order to start and stop services, when on my other boxes I would authenticate as my currently logged in user. Normally I like to run sudo -i or sudo -s in order to run multiple tasks in the same session, and this was never previously a problem until I recently locked down the root account such that I could no longer enter its password (it no longer had one).

On Gentoo at least, when you need to execute some action on some service, the usage of run_init is required. The purpose of this utility is to change SELinux contexts from a user context to a system context for the purpose of running some action on some service. The reason for this is that doing this requires a change to the SELinux system user, role, and type, a highly privileged action. You can read more about this here. With this in mind, I started digging through my SELinux configuration, looking for anything out of the ordinary. Everything was in place. My SELinux login mapping was correct and I had access to all the correct roles. Then, I started looking through PAM. run_init is configured to authenticate through PAM using the configuration in /etc/pam.d/run_init which had no differences between both of my systems. Also checking /etc/pam.d/system-auth turned up no differences.

At this point I was beginning to pull my hair out a little, and went down the path of running strace against run_init to see what exactly was going on. I noticed that on the system where run_init was working as I was expecting, by asking me for my credentials instead of root’s, run_init was reading the contents of /proc/self/loginuid, and the other system was not. Immediately after this file was read, PAM was invoked using the UID present in this file. That’s when I had a vague realization of what was happening.

I did a quick search for what package was providing the run_init binary and thought to check its USE flags.

# equery b `which run_init`
 * Searching for /usr/sbin/run_init ...
sys-apps/policycoreutils-3.1 (/usr/sbin/run_init)

# equery u policycoreutils
[ Legend : U - final flag setting for installation]
[ : I - package is installed with flag ]
[ Colors : set, unset ]
 * Found these USE flags for sys-apps/policycoreutils-3.1:
 U I
 - - audit : Enable support for sys-process/audit and use the audit_* functions (like audit_getuid instead of
 getuid())
 - - dbus : Enable dbus support for anything that needs it (gpsd, gnomemeeting, etc)
 + + pam : Add support for PAM (Pluggable Authentication Modules) - DANGEROUS to arbitrarily flip
 - - python_targets_python3_6 : Build with Python 3.6
 + + python_targets_python3_7 : Build with Python 3.7

Ah ha! Sure enough, on the system where my user was being authenticated, the audit USE flag was set, and here it wasn’t. A quick flip of this flag and a rebuild of policycoreutils fixed the problem.

This was a little bit of an oddity at first. I feel like this functionality should be separated from the audit USE flag in some way, as it looks like run_init checks what user is invoking some action in order to log it, but this information is also passed along to PAM. Without this check, run_init seems to assume that the root user is the one performing the action and does not authenticate the actual user against PAM. In any case, this simple change implemented the exact behavior I was looking for.

Quick Word on Gitea Dump

Sat, 18 Jul 2020 22:04:15 -0400

Only today did I notice that my Gitea instance was unable to create new repositories. A furious search turned up this issue regarding the issue. This comment explains it in more detail. The fix is to run this on the Gitea database:

SELECT SETVAL('public.access_id_seq', COALESCE(MAX(id), 1) ) FROM public.access;
SELECT SETVAL('public.access_token_id_seq', COALESCE(MAX(id), 1) ) FROM public.access_token;
SELECT SETVAL('public.action_id_seq', COALESCE(MAX(id), 1) ) FROM public.action;
SELECT SETVAL('public.attachment_id_seq', COALESCE(MAX(id), 1) ) FROM public.attachment;
SELECT SETVAL('public.collaboration_id_seq', COALESCE(MAX(id), 1) ) FROM public.collaboration;
SELECT SETVAL('public.comment_id_seq', COALESCE(MAX(id), 1) ) FROM public.comment;
SELECT SETVAL('public.commit_status_id_seq', COALESCE(MAX(id), 1) ) FROM public.commit_status;
SELECT SETVAL('public.deleted_branch_id_seq', COALESCE(MAX(id), 1) ) FROM public.deleted_branch;
SELECT SETVAL('public.deploy_key_id_seq', COALESCE(MAX(id), 1) ) FROM public.deploy_key;
SELECT SETVAL('public.email_address_id_seq', COALESCE(MAX(id), 1) ) FROM public.email_address;
SELECT SETVAL('public.follow_id_seq', COALESCE(MAX(id), 1) ) FROM public.follow;
SELECT SETVAL('public.gpg_key_id_seq', COALESCE(MAX(id), 1) ) FROM public.gpg_key;
SELECT SETVAL('public.hook_task_id_seq', COALESCE(MAX(id), 1) ) FROM public.hook_task;
SELECT SETVAL('public.issue_assignees_id_seq', COALESCE(MAX(id), 1) ) FROM public.issue_assignees;
SELECT SETVAL('public.issue_dependency_id_seq', COALESCE(MAX(id), 1) ) FROM public.issue_dependency;
SELECT SETVAL('public.issue_id_seq', COALESCE(MAX(id), 1) ) FROM public.issue;
SELECT SETVAL('public.issue_label_id_seq', COALESCE(MAX(id), 1) ) FROM public.issue_label;
SELECT SETVAL('public.issue_user_id_seq', COALESCE(MAX(id), 1) ) FROM public.issue_user;
SELECT SETVAL('public.issue_watch_id_seq', COALESCE(MAX(id), 1) ) FROM public.issue_watch;
SELECT SETVAL('public.label_id_seq', COALESCE(MAX(id), 1) ) FROM public.label;
SELECT SETVAL('public.lfs_lock_id_seq', COALESCE(MAX(id), 1) ) FROM public.lfs_lock;
SELECT SETVAL('public.lfs_meta_object_id_seq', COALESCE(MAX(id), 1) ) FROM public.lfs_meta_object;
SELECT SETVAL('public.login_source_id_seq', COALESCE(MAX(id), 1) ) FROM public.login_source;
SELECT SETVAL('public.milestone_id_seq', COALESCE(MAX(id), 1) ) FROM public.milestone;
SELECT SETVAL('public.mirror_id_seq', COALESCE(MAX(id), 1) ) FROM public.mirror;
SELECT SETVAL('public.notice_id_seq', COALESCE(MAX(id), 1) ) FROM public.notice;
SELECT SETVAL('public.notification_id_seq', COALESCE(MAX(id), 1) ) FROM public.notification;
SELECT SETVAL('public.org_user_id_seq', COALESCE(MAX(id), 1) ) FROM public.org_user;
SELECT SETVAL('public.protected_branch_id_seq', COALESCE(MAX(id), 1) ) FROM public.protected_branch;
SELECT SETVAL('public.public_key_id_seq', COALESCE(MAX(id), 1) ) FROM public.public_key;
SELECT SETVAL('public.pull_request_id_seq', COALESCE(MAX(id), 1) ) FROM public.pull_request;
SELECT SETVAL('public.reaction_id_seq', COALESCE(MAX(id), 1) ) FROM public.reaction;
SELECT SETVAL('public.release_id_seq', COALESCE(MAX(id), 1) ) FROM public.release;
SELECT SETVAL('public.repo_indexer_status_id_seq', COALESCE(MAX(id), 1) ) FROM public.repo_indexer_status;
SELECT SETVAL('public.repo_redirect_id_seq', COALESCE(MAX(id), 1) ) FROM public.repo_redirect;
SELECT SETVAL('public.repo_unit_id_seq', COALESCE(MAX(id), 1) ) FROM public.repo_unit;
SELECT SETVAL('public.repository_id_seq', COALESCE(MAX(id), 1) ) FROM public.repository;
SELECT SETVAL('public.review_id_seq', COALESCE(MAX(id), 1) ) FROM public.review;
SELECT SETVAL('public.star_id_seq', COALESCE(MAX(id), 1) ) FROM public.star;
SELECT SETVAL('public.stopwatch_id_seq', COALESCE(MAX(id), 1) ) FROM public.stopwatch;
SELECT SETVAL('public.team_id_seq', COALESCE(MAX(id), 1) ) FROM public.team;
SELECT SETVAL('public.team_repo_id_seq', COALESCE(MAX(id), 1) ) FROM public.team_repo;
SELECT SETVAL('public.team_unit_id_seq', COALESCE(MAX(id), 1) ) FROM public.team_unit;
SELECT SETVAL('public.team_user_id_seq', COALESCE(MAX(id), 1) ) FROM public.team_user;
SELECT SETVAL('public.topic_id_seq', COALESCE(MAX(id), 1) ) FROM public.topic;
SELECT SETVAL('public.tracked_time_id_seq', COALESCE(MAX(id), 1) ) FROM public.tracked_time;
SELECT SETVAL('public.two_factor_id_seq', COALESCE(MAX(id), 1) ) FROM public.two_factor;
SELECT SETVAL('public.u2f_registration_id_seq', COALESCE(MAX(id), 1) ) FROM public.u2f_registration;
SELECT SETVAL('public.upload_id_seq', COALESCE(MAX(id), 1) ) FROM public.upload;
SELECT SETVAL('public.user_id_seq', COALESCE(MAX(id), 1) ) FROM public."user";
SELECT SETVAL('public.user_open_id_id_seq', COALESCE(MAX(id), 1) ) FROM public.user_open_id;
SELECT SETVAL('public.version_id_seq', COALESCE(MAX(id), 1) ) FROM public.version;
SELECT SETVAL('public.watch_id_seq', COALESCE(MAX(id), 1) ) FROM public.watch;
SELECT SETVAL('public.webhook_id_seq', COALESCE(MAX(id), 1) ) FROM public.webhook;

Welcome

Sat, 18 Jul 2020 18:37:00 -0400

And so as of the time of writing this, my blog is now officially live. I am still adding content to my other pages and will soon add a links page, but for now I leave any readers with this notice and a wide future of upcoming content. I also anticipate uploading the full source of the site to a public Git repository.

Until next time, see ya.

Home on ./concord.sh

Blog

Links

madaidan’s Insecurities

GrapheneOS

About

Writing a Kubernetes Operator to Fix IPv6

A Small Follow-up to UEFI Secure Boot

PCR 7 and Microsoft’s Third-Party/Vendor CA Certificate

Microsoft’s Third-Party/Vendor CA Certificate May Be Required on Some Hardware

Can malware touch secure boot properties while the OS is running?

sbctl, the Secure Boot Key Manager

UEFI Secure Boot the Right Way

The Current Situation

The Solution

Generating Secure Boot Signing Keys

Installing Keys and Signing Things

Putting It All Together

Going a Step Further

"Jammy Jellyfish" jammed my girlfriend's laptop

How not carefully reading the docs cost me a heart attack and a change of pants

License and Copyright

Gentoo Clang Toolchain Initial Setup

Nextcloud cron with SELinux

Wrong run_init Authentication

Quick Word on Gitea Dump

Welcome

`sbctl`, the Secure Boot Key Manager