Cross-Compiling NixOS images
This is the fourth and final part of our four-part article series in which we have a closer look at how easy it is to create GNU/Linux systems with NixOS that are:
- Small: no unnecessary parts, built with Nix, but contains no Nix itself.
- NixOS is not designed to be small up front, but provides great facilities to make system configurations and packages minimal.
- We will use
systemd-repartto create minimal raw system images.
- Self-inflating: Hardware disk size is not always known at image build time, but the image can inflate itself at the first boot to use the whole disk.
- We will again use
systemd-repartto repartition the disk at early boot time.
- We will again use
- Auto-updating:
systemd-sysupdateprovides powerful mechanisms to make systems self-updatable over network.- NixOS already provides system switching capabilities. These necessitate Nix being installed on that system to manage system generations, which is not what we want on a minimal appliance.
systemd-sysupdatehelps us create and swap immutable A/B system partitions instead
In the different parts of the article series, we handle the following topics step by step:
- NixOS appliance images with systemd-repart
- Minimizing NixOS images
- Immutable A/B system partitions with NixOS for over-the-air updates with systemd-sysupdate
- Cross-compiling the image for other platforms (👈 this article)
Defining NixOS configurations in Nix Flakes
Although this article is about cross-compiling the NixOS appliance image from the last articles, let’s start from a fresh state with a few basics and then develop that further towards cross-compilation of the original image. This way it will be more comprehensible.
In a new repository, we can start defining a system configuration called image inside the nixosConfigurations output category of a flake:
# File: flake.nix
{
inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
outputs = inputs: {
nixosConfigurations.image = inputs.nixpkgs.lib.nixosSystem {
modules = [
./system-configuration/configuration.nix
];
};
};
}The configuration.nix file defines the actual system configuration.
For this example, we took the configuration from the last article and simplified it a bit because this article and its example repo are really about cross-compilation, not full-blown images.
You find all the files here in the example repository.
We reference it as a module because this way Flake things and NixOS system things are separated from each other, but this is, of course, optional.
To test if the configuration even evaluates, we can build the top-level derivation:
$ nix build .#nixosConfigurations.image.config.system.build.toplevel
# alternatively:
$ nixos-rebuild build --flake .#imageWe know this from the second article about minimizing images, where we also learned how to inspect the system configuration after loading the flake in the REPL:
$ nix repl .
nix-repl> nixosConfigurations.image.config.systemd.package.version
"258.2"
nix-repl> :p nixosConfigurations.image.options.networking.hostName.definitionsWithLocations
[
{
file = "/nix/store/3860s11qbjcaxn7q1rf1l2gc280zxrgr-source/system-configuration/configuration.nix";
value = "crosscomp-example";
}
]By putting any available system configuration into the nixosConfigurations attribute category first, we keep it available for all kinds of inspection.
Building images from nixosConfigurations in Flakes
Now, let’s build an actual image from it because the toplevel derivation is not useful for image deployments.
# file: flake.nix
{
inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
outputs = inputs: {
nixosConfigurations.image = inputs.nixpkgs.lib.nixosSystem {
modules = [
./system-configuration/configuration.nix
];
};
packages.x86_64-linux = {
image = inputs.self.nixosConfigurations.image.config.system.build.image;
};
};
}The output image in the packages category now references the image output of the system configuration.
We learned how to define these in the first article of this series about creating images with systemd-repart.
The details for this image are in the file system-configuration/image.nix.
(If you found this article first and are interested to learn how to configure images, we suggest to have a look at the other articles as we concentrate on cross-compilation in this one.)
Now, we can run one command to create the bootable image:
$ nix build .#imageIt’s great when this works, but this is still not the cross-compilation that we are here for.
Cross-compiling images
Assuming that we’re building from an x86_64-linux and want to cross-compile an aarch64-linux image, we can now write the following:
# file: flake.nix
{
inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
outputs = inputs: {
nixosConfigurations.image = inputs.nixpkgs.lib.nixosSystem {
modules = [
./system-configuration/configuration.nix
];
};
packages.x86_64-linux = {
image = inputs.self.nixosConfigurations.image.config.system.build.image;
image-aarch64 = (inputs.self.nixosConfigurations.image.extendModules {
modules = [
{
nixpkgs.buildPlatform = "x86_64-linux";
nixpkgs.hostPlatform = "aarch64-linux";
}
];
}).config.system.build.image;
};
};
}The new package attribute image-aarch64 references the same nixosConfigurations.image attribute, but picks the extendModules function.
This function allows us to inject additional NixOS modules into the system configuration.
To make the normal image a cross-build image, we can manipulate the platform configuration values in the injected NixOS module:
nixpkgs.buildPlatform: Specifies the platform on which NixOS should be built.nixpkgs.hostPlatform: Specifies the platform where the NixOS configuration will run.
If these values are different, the build becomes a cross-build!
(We are not getting an error for setting the nixpkgs.hostPlatform value twice because we made the first definition of this value overridable in configuration.nix)
We can now cross-compile the image for aarch64-linux by running:
$ nix build .#image-aarch64Building any platform from any platform
We have now seen how to build for platform Y on platform X (with Y=aarch64-linux and X=x86_64-linux).
But what if we want to build on platform X for platform Y?
And if we add another target platform, like e.g., riscv64-linux, and want to build that on X or Z?
Building on one platform for one other
Let’s first write a dumb and repetitive version and then fix it:
# file: flake.nix
{
inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
outputs = inputs: {
# ...
packages.x86_64-linux = {
# ...
image-aarch64 = (inputs.self.nixosConfigurations.image.extendModules {
modules = [
{
nixpkgs.buildPlatform = "x86_64-linux";
nixpkgs.hostPlatform = "aarch64-linux";
}
];
}).config.system.build.image;
};
# (A) evaluating platform
packages.aarch64-linux = {
image-x86_64 = (inputs.self.nixosConfigurations.image.extendModules {
modules = [
{
nixpkgs.buildPlatform = "aarch64-linux"; # (B) build platform
nixpkgs.hostPlatform = "x86_64-linux"; # (C) target platform
}
];
}).config.system.build.image;
};
};
}This is exactly the repetition that we don’t want long-term, but it’s important to understand all the different system strings at different locations in the flake here. Let’s look at the different spots as commented in the code:
(A) packages.XXX:
The attribute name after the packages output category defines the platform that evaluates the system.
This means that if we are on platform aarch64-linux and run nix build .#foo, it will evaluate and build the flake output packages.aarch64-linux.foo.
(B) and (C) define the building platform and the booting platform, but we already discussed that.
A note about a common Nix Flakes-related misconception:
Changing the XXX in packages.XXX.myPackageName does not help with cross-compiling anything.
The cross-compilation magic is all in the NixOS configuration.
Running nix flake show now enumerates the goals for each evaluating platform:
$ nix flake show
├───nixosConfigurations
│ └───image: NixOS configuration
└───packages
├───aarch64-linux
│ └───image-x86_64 omitted (use '--all-systems' to show)
└───x86_64-linux
├───image: package 'crosscompilation-image-example'
└───image-aarch64: package 'crosscompilation-image-example'We see that from an aarch64-linux system, we could build the image for 86_64-linux.
Also, from an x86_64-linux, we can build the image for aarch64-linux.
Generalizing cross-platform builds
We just wrote and understood the code to build an image for one platform from another.
In these image descriptions, the NixOS configuration attributes nixpkgs.buildPlatform and nixpkgs.hostPlatform simply differ, and the rest happens automagically.
The image attribute in the x86_64-linux platform list doesn’t have buildPlatform set, which makes it a native image.
Of course, as Nix is not JSON/YAML but a full scripting language, we can automate cross-compilation from build platform X for running on platform Y by defining lists of these platforms and then mapping over those:
# file: flake.nix
# ...
outputs = inputs:
let
inherit (inputs.nixpkgs) lib;
buildPlatforms = [
"x86_64-linux"
"aarch64-linux"
];
# (A): Accept a function that maps a system string to a set of packages.
# Return the package set for every platform.
forEachSystem = lib.genAttrs buildPlatforms;
# (B): Accept a tuple of build and host platforms.
# Return the image derivation.
imageFunction =
{ hostPlatform, buildPlatform }:
(inputs.self.nixosConfigurations.image.extendModules {
modules = [
{
nixpkgs = { inherit hostPlatform buildPlatform; };
}
];
}).config.system.build.image;
in
{
nixosConfigurations.image = inputs.nixpkgs.lib.nixosSystem {
modules = [
./system-configuration/configuration.nix
];
};
# (C): Define `packages.XXX = { ... }` for each buildPlatform
packages = forEachSystem (system: {
# (D): Define image variants based on build/host platform differences
image-aarch64 = imageFunction {
buildPlatform = system;
hostPlatform = "aarch64-linux";
};
image-x86_64 = imageFunction {
buildPlatform = system;
hostPlatform = "x86_64-linux";
};
});
};We just added quite a few lines of helper code to generalize the pattern from earlier:
- (A): The
forEachSystemfunction accepts a function likesystem: { }that returns a set. This set is meant to contain package definitions forsystem.forEachSystemwill then call it for each system that we defined as a possible build platform. (This is similar toflake-utils, and Marijan wrote a blog post earlier this year on why you don’t need it because it’s basically a one-liner) - (B): As we have already seen earlier, reinstating the same image definition multiple times with different build/host platforms is very repetitive.
For this reason, we add the helper function
imageFunctionthat accepts the build/host platform tuple and returns the image, hiding the boilerplate code. - (C): We assign the return set of
forEachSystem (system: { ...} )to the top-levelpackagesoutput set. This creates the images in our attribute set for each build platform, which can also evaluate the flake. - (D): Calling our
imageFunctionwith different host and build platforms looks very tidy now!
The nix flake show output shows that we can now build each platform from each system:
$ nix flake show
├───nixosConfigurations
│ └───image: NixOS configuration
└───packages
├───aarch64-linux
│ ├───image-aarch64 omitted (use '--all-systems' to show)
│ └───image-x86_64 omitted (use '--all-systems' to show)
└───x86_64-linux
├───image-aarch64: package 'crosscompilation-image-example'
└───image-x86_64: package 'crosscompilation-image-example'Adding macOS
Feel free to skip this section if you don’t use macOS.
It is possible to build Linux systems on macOS. To do that, we only have to set up the macOS Linux builder. We describe how to do this in this blog post.
To enable our flake for builds that run on the Linux builder on macOS, we just need to add aarch64-darwin as a build platform and another small helper function:
# file: flake.nix
# ...
outputs =
inputs:
let
inherit (inputs.nixpkgs) lib;
# (A): Helper that transforms "xxx-darwin" to "xxx-linux"
toLinux = builtins.replaceStrings [ "darwin" ] [ "linux" ];
buildPlatforms = [
"x86_64-linux"
"aarch64-linux"
# (B): Add ARM macOS
"aarch64-darwin"
];Adding aarch64-darwin to the build platforms adds another attribute set for darwin in the packages top-level output platforms:
$ nix flake show
path:/tmp/test?lastModified=1766231693&narHash=sha256-WPiYQ8wFxR5cEF6avF0DNjHPY3fzFKZT3a9lhV/7IdQ%3D
├───nixosConfigurations
│ └───image: NixOS configuration
└───packages
├───aarch64-darwin
│ ├───image-aarch64 omitted (use '--all-systems' to show)
│ └───image-x86_64 omitted (use '--all-systems' to show)
├───aarch64-linux
│ ├───image-aarch64 omitted (use '--all-systems' to show)
│ └───image-x86_64 omitted (use '--all-systems' to show)
└───x86_64-linux
├───image-aarch64: package 'crosscompilation-image-example'
└───image-x86_64: package 'crosscompilation-image-example'The only missing bit is to install the toLinux helper function.
We set the aarch64-darwin system as the buildPlatform, but this platform cannot really build all the Linux packages.
(It’s generally possible, but there is not enough support for that at this point. Many packages would break.)
However, on macOS, we only need the darwin Nix to evaluate our package expressions.
The evaluated derivation descriptions will then be delegated to the Linux builder VM, which runs as a background service.
For that, we just need to swap the XXX-darwin platform strings to XXX-linux, and that’s it:
# file: flake.nix
# ...
packages = forEachSystem (system: {
image-aarch64 = imageFunction {
# (C): Transform potential darwin to linux substrings
buildPlatform = toLinux system;
hostPlatform = "aarch64-linux";
};
image-x86_64 = imageFunction {
buildPlatform = toLinux system;
hostPlatform = "x86_64-linux";
};
});The builds on macOS are relatively fast. It’s only that there is a bit of overhead from file transfer between the darwin Nix daemon and the builder VM.
Bonus: Building all images for all platforms on all platforms
In the examples before, we still had to write out one output attribute for each image and target platform. Let’s automate this away and generate the Cartesian product of all target platforms and NixOS configurations in the flake for all build platforms:
# file: flake.nix
{
inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
outputs =
inputs:
let
inherit (inputs.nixpkgs) lib;
toLinux = builtins.replaceStrings [ "darwin" ] [ "linux" ];
buildPlatforms = [
"x86_64-linux"
"aarch64-linux"
"aarch64-darwin"
];
# (A) This is the list of all the targets without "-linux" suffix
targetPlatforms = [
"aarch64"
"riscv64"
"x86_64"
];
forEachSystem = lib.genAttrs buildPlatforms;
# (B) the image function now also accepts the image as input
# so we can transform different images to cross-compilation targets
imageFunction =
{
image,
hostPlatform,
buildPlatform,
}:
(image.extendModules {
modules = [
{
nixpkgs = { inherit hostPlatform buildPlatform; };
}
];
}).config.system.build.image;
in
{
# (C) We add multiple NixOS configurations
# with slightly different content
nixosConfigurations = {
foo = inputs.nixpkgs.lib.nixosSystem {
modules = [
./system-configuration/configuration.nix
{ networking.hostName = "foo"; }
];
};
bar = inputs.nixpkgs.lib.nixosSystem {
modules = [
./system-configuration/configuration.nix
{ networking.hostName = "bar"; }
];
};
};
packages = forEachSystem (
system:
let
# (D) for each given combination of:
# - image name (as in inputs.self.nixosConfigurations.XXX)
# - target platform string (as in "x86_64" without "-linux")
# - but only for the given build platform, as we are already
# in the `forEachSystem` context
# We generate an image derivation.
# This is, however, returned in this form:
# { name = "imagename"; value = <image derivation>; }
# because we map it over lib.listToAttrs later
f =
{
imageName,
targetPlatformString,
buildPlatform,
}:
{
# this will be the `packages.XXX.imageName-platform` attribute
name = imageName + "-" + targetPlatformString;
value = imageFunction {
image = inputs.self.nixosConfigurations.${imageName};
buildPlatform = toLinux buildPlatform;
hostPlatform = targetPlatformString + "-linux";
};
};
in
lib.listToAttrs (
# (E) mapCartesianProduct creates all combinations of
# image names and target platforms here
lib.mapCartesianProduct f {
imageName = builtins.attrNames inputs.self.nixosConfigurations;
buildPlatform = [ (toLinux system) ];
targetPlatformString = targetPlatforms;
}
)
);
};
}- (A): Instead of writing
image-x86_64 = ...andimage-aarch64 = ...etc., we define a list of platforms. - (B): We continue to use the
imageFunctionhelper, but let it accept different images. - (C): Instead of cross-compiling just one image, we demo how to do it for any number of images as long as they are listed under
nixosConfigurations. - (D): Function
faccepts the image attribute name, the target platform string without-linuxsuffix, and the build platform, and then returns a{ name = ...; value = ...; }attribute. This form is important because we transform a list of these into the packages attribute set later usinglistToAttrs. - (E): The
mapCartesianProductfunction creates the Cartesian set of input combinations as a list. We map this list over our functionfand then convert it to an attribute set that finally becomes ourpackagesoutput.
Now we can build all target variants of all images on all systems:
$ nix flake show
├───nixosConfigurations
│ ├───bar: NixOS configuration
│ └───foo: NixOS configuration
└───packages
├───aarch64-darwin
│ ├───bar-aarch64 omitted (use '--all-systems' to show)
│ ├───bar-riscv64 omitted (use '--all-systems' to show)
│ ├───bar-x86_64 omitted (use '--all-systems' to show)
│ ├───foo-aarch64 omitted (use '--all-systems' to show)
│ ├───foo-riscv64 omitted (use '--all-systems' to show)
│ └───foo-x86_64 omitted (use '--all-systems' to show)
├───aarch64-linux
│ ├───bar-aarch64 omitted (use '--all-systems' to show)
│ ├───bar-riscv64 omitted (use '--all-systems' to show)
│ ├───bar-x86_64 omitted (use '--all-systems' to show)
│ ├───foo-aarch64 omitted (use '--all-systems' to show)
│ ├───foo-riscv64 omitted (use '--all-systems' to show)
│ └───foo-x86_64 omitted (use '--all-systems' to show)
└───x86_64-linux
├───bar-aarch64: package 'crosscompilation-image-example'
├───bar-riscv64: package 'crosscompilation-image-example'
├───bar-x86_64: package 'crosscompilation-image-example'
├───foo-aarch64: package 'crosscompilation-image-example'
├───foo-riscv64: package 'crosscompilation-image-example'
└───foo-x86_64: package 'crosscompilation-image-example'Potential anti-pattern: Overriding pkgs
Instead of setting nixpkgs.hostPlatform and nixpkgs.buildPlatform, we could also override nixpkgs.pkgs like this:
image.extendModules {
modules = [
{
nixpkgs.pkgs = import inputs.nixpkgs {
localSystem = A;
crossSystem = B;
};
}
];
}).config.system.build.imageThis variant of doing it is not wrong and seems more straightforward than what we did in this article so far, as the nixpkgs documentation also shows how to cross-compile individual packages like this.
For NixOS systems, however, we suggest using the nixpkgs.hostPlatform and nixpkgs.buildPlatform way.
The reason for this is that nixpkgs.pkgs (search.nixos.org documentation) is automatically created from the other configuration paths in the NixOS configuration.
This includes overlay definitions and other settings that may be spread over different NixOS modules - these would all be ignored when setting nixpkgs.pkgs as a whole, reducing the composability of our NixOS modules.
Summary and outlook
If you don’t have a lot of experience with Nix(OS) just yet and still wonder why Nix doesn’t use JSON/YAML to describe packages and systems, this article might be a bit overwhelming. But if you look past that, you will realize that systems that use JSON/YAML force us to automate the generation of such structures. In Nix, we can script anything because we are not limited to primitive data formats.
We created a flake that:
- Defines one or multiple NixOS configurations in the top-level
nixosConfigurationattribute. This enables us to query information from the NixOS configuration, build VMs instead of images, etc. - References the NixOS configuration in the
packagesoutput, injects additional config attributes into it, and then selects specific output formats. This way, we can provide cross-compilation targets and/or different image outputs from the same NixOS configurations.
This structure makes it very easy to add more images and more platforms without changing the NixOS configuration of the systems themselves. Separating the system configurations and the cross-compilation configs keeps all the doors open for testing, porting, and adding more variants or platforms.
The image config that we used in this article is simple. Real-life projects might be more complex because more complex packages and services might have their own quirks that would need to be handled differently per platform. This is also no problem with Nixpkgs overlays, which we can inject when needed!
To have the CI build all the image variants all the time, we could, for example, set checks = packages; in the outputs attribute set and then run nix flake check in the CI.
The final repository is here on GitHub:
- The repository that only highlights the cross-compilation aspects is here: https://github.com/applicative-systems/nixos-image-cross-compilation-example
- Original repository with the appliance image: https://github.com/applicative-systems/simple-systemd-repart-nixos-image
This completes our 4-part blog article series. We would love to know if you found it useful and what you are building with NixOS and systemd! Drop us a mail: hello@nixcademy.com
We help many customers transfer from other Linux-based solutions to NixOS or improve their existing NixOS-based solutions. From that experience, we can help you copy the successful patterns of winning organizations and avoid the patterns that have not worked well elsewhere, instead of having to make this experience yourself from scratch. No matter if you just need a quick consultation on how to build something with Nix or if we can help you by lending developer time, schedule a quick call with us or e-mail us.
