C++ with Nix in 2023, Part 2: Package Generation and Cross-Compilation

📆 Thu Nov 16 2023 by Jacek Galowicz

(11 min. reading time)

This is Part 2 of our series on using C++ with Nix in 2023 we’re moving a step further to focus on packaging and cross-compilation. Packaging is crucial for making your applications easily distributable, while cross-compilation allows you to build software for different platforms from a single source.

In part 1 of this series, we explored the basics of setting up a Nix shell for C++ development. This week, we learn how to create a Nix package from our example project.

A package definition in Nix allows for sandboxed single-command package builds that are truly reproducible, even on computers of non-developers who happen to have Nix installed. With a Nix package definition, we also get the following other goodies:

Developer shells can be automatically generated from package definitions. So we can ditch our old manual development shell definition to avoid duplicate efforts.
Nix flakes provide a nice way for users to run our application without even checking out the code.
Assuming portable code and build system setup, Nix gives us cross-compilation for free!
It is also easy to swap compilers after the fact. This is especially interesting if you want to regularly check the compatibility of your code with multiple new and old compilers.

The final code for this article is online in the same GitHub repo as the first article, but on branch part2: https://github.com/tfc/cpp-nix-2023/tree/part2

Create a Package for Our App

Until now, we got to know the nix develop use case for developers who need all toolchain and dependency packages in their shell environment to work on the project. This is already better than Docker images or VMs because we don’t need to mount the source code into containers or similar. Our favorite text editors also have access to tools and libs when we start them from a Nix shell (We can in fact start as many different Nix shells per project as we like).

However, consumers of our apps and libraries don’t care about our development environment. They simply want to have the package for consumption. Wouldn’t it be great if they could simply run nix build and get it without even having to know what language it is written in or how to set up the shell and run the build?

Let’s do that by first creating a package.nix file that describes the package. The code is very short, but I added many comments for comprehensibility:

# file: package.nix

# The items in the curly brackets are function parameters as this is a Nix
# function that accepts dependency inputs and returns a new package
# description
{ lib
, stdenv
, cmake
, boost
, catch2
}:

# stdenv.mkDerivation now accepts a list of named parameters that describe
# the package itself.

stdenv.mkDerivation {
  name = "cpp-nix";

  # good source filtering is important for caching of builds.
  # It's easier when subprojects have their own distinct subfolders.
  src = lib.sourceByRegex ./. [
    "^src.*"
    "^test.*"
    "CMakeLists.txt"
  ];

  # We now list the dependencies similar to the devShell before.
  # Distinguishing between `nativeBuildInputs` (runnable on the host
  # at compile time) and normal `buildInputs` (runnable on target
  # platform at run time) is an important preparation for cross-compilation.
  nativeBuildInputs = [ cmake ];
  buildInputs = [ boost catch2 ];

  # Instruct the build process to run tests.
  # The generic builder script of `mkDerivation` handles all the default
  # command lines of several build systems, so it knows how to run our tests.
  doCheck = true;
}

This package description contains no information about how to run the configuration step, how to build the binaries, how to run the tests, and how to finally install the cpp-nix app. The mkDerivation function automagically knows it. But the process is not magical: Making our package depend on cmake also installs certain hooks that educate mkDerivation’s builder script with the knowledge of how to deal with CMake projects.

If our CMake project definition was highly customized so that it deviates from standard projects too much, these customizations could be added to the package definition. Many C++ projects have ill-formed CMake/Meson/autotools build systems, which increases the effort on the package maintainer side, but this is nothing new to nixpkgs.

We can now “call” the package from the flake and also let the dev shell reuse it:

# file: flake.nix
# ...

  perSystem = { config, self', inputs', pkgs, system, ... }: {
    # The `callPackage` automatically fills the parameters of the function
    # in package.nix with what's inside the `pkgs` attribute.
    packages.default = pkgs.callPackage ./package.nix { };

    # The `config` variable contains our own outputs, so we can reference
    # neighbor attributes like the package we just defined one line earlier.
    devShells.default = config.packages.default;
  };

# ...

The callPackage function fills out all parameters from the dependency list at the beginning of our package.nix file. This way the package.nix file describes the mechanism of how to build the package, while the flake.nix file contains the policy decisions of where to get which package from (in this case: which nixpkgs pin to use). This is also the point where we could override the compiler or library inputs.

By assigning the package to our devShells.default attribute, the nix develop command simply takes all dependency information from there, which perfectly works for smaller repos. Monorepos with multiple projects work a bit differently, but this article doesn’t deal with that.

Let’s build it. Note that we don’t need the nix develop shell for this:

$ nix build
$ ./result/bin/cpp-nix
Hello World!
Compiler: g++ 12.3.0
Boost: 1.81.0

There is no trace of a build folder or anything, which we get when we compile the code manually in the developer shell. Nix creates a sandbox environment from scratch, builds the package inside, and then throws away everything but the package output. We only get a symlink with the name result in the current folder which points to the /nix/store/...cpp-nix/ path that contains our package outputs.

We can also directly run it in a single command without having to build and run in two steps:

$ nix run
Hello World!
Compiler: g++ 12.3.0
Boost: 1.81.0

Note that nix automatically finds out if it needs to rebuild the package based on content changes in the project folder.

We can even build and run it from GitHub without having to clone the repository first:

$ nix run github:tfc/cpp-nix-2023
Hello World!
Compiler: g++ 12.3.0
Boost: 1.81.0

At this point, there is practically no difference in the user experience between “real” Nix packages and ours.

Swap the Compiler

We already did that with the developer shell environment in the last article, but on a package, it’s even a bit more straight-forward: The packages that we list in package.nix’s first lines as function inputs can all be overridden with a different choice than the defaults. Swapping the compiler now looks like this:

# file: flake.nix
# ...

  packages = {
    default = pkgs.callPackage ./package.nix { };
    clang = pkgs.callPackage ./package.nix { stdenv = pkgs.clangStdenv; };
  };

# ...

This new attribute can be built with nix build .#clang. A CI could now always build both variants so that developers can keep supporting multiple compiler toolchains.

On GNU/Linux, Nix’s default C++ compiler is GCC, while it’s clang on macOS. Use pkgs.clangStdenv or pkgs.gccStdenv respectively.

Cross-Compilation

There is nothing inherently architecture-specific about our little hello-world app. To cross-compile it, we need a cross-compiler and the Boost library for the target platform. We would also need to disable the tests because even if we can cross-compile them, we would usually not be able to run them on the same host.

Let’s first make testing optional in our package.nix file:

# file: package.nix
{ lib
, stdenv
, cmake
, boost
, catch2
# Tests are by default enabled, but the caller can disable them
, enableTests ? true
}:

# ...

  # The third category `checkInputs` is ignored if tests are disabled
  nativeBuildInputs = [ cmake ];
  buildInputs = [ boost ];
  checkInputs = [ catch2 ];

  doCheck = enableTests;

  # Our CMakeLists.txt has already been prepared from the beginning
  cmakeFlags = lib.optional (!enableTests) "-DTESTING=off";
}

We will later see how to call this package with disabled tests.

Selecting the cross-compiler is generally done by creating a different stdenv which not only contains the right compiler but also selects the right libc and other toolchain settings. In addition to that, all the dependency libraries that are needed for/by the target platform, also need to be exchanged.

While this could be a lot of work for big projects, Nixpkgs makes this really simple by providing a pkgs.pkgsCross.XXX attribute that contains all the pkgs for architecture XXX.

In the following, we use aarch64-multiplatform for cross-compiling from systems from non-ARM systems to ARM platforms and gnu64 to build packages for Intel-based platforms. macOS users would select aarch64-darwin and x86_64-linux-darwin. We will have a little outlook on what else is there in the following subsection.

Our next changes on the flake look like this:

# file: flake.nix
# ...

    packages = {
      default = pkgs.callPackage ./package.nix { };
    } // pkgs.lib.optionalAttrs (system != "x86_64-linux") {
      crossIntel = pkgs.pkgsCross.gnu64.callPackage ./package.nix {
        enableTests = false;
      };
    } // pkgs.lib.optionalAttrs (system != "aarch64-linux") {
      crossAarch64 = pkgs.pkgsCross.aarch64-multiplatform.callPackage ./package.nix {
        enableTests = false;
      };
    };

# macOs users exchange the `-linux` suffix for `-darwin`

# ...

What we are doing here is populating the packages attribute set with more packages. On non-Intel systems, we add the crossIntel package and on non-ARM systems, we add the crossAarch64 package.

The // operator can be used to merge two attribute sets (technically, it updates/overwrites the attributes of its left operand with the ones of its right operand - see docs)

The optionalAttrs function accepts a bool condition variable and an attribute set as parameters. If the condition is true, it returns the whole attribute. Otherwise, it returns an empty attribute set. Merging one attribute set with an empty one has no effect of course.

Now, we can build the respective cross-package. Assuming we are on an Intel-based Linux machine, we would use crossArch64:

$ nix build .#crossAarch64
$ file ./result/bin/cpp-nix
./result/bin/cpp-nix: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /nix/store/2hy50jbh5bfdn4ial7j7lrr5r7byavgw-glibc-aarch64-unknown-linux-gnu-2.38-23/lib/ld-linux-aarch64.so.1, for GNU/Linux 3.10.0, not stripped

The general experience with cross-compilation of packages on Nix is very painless, but only with packages that are truly portable. Many C/C++ projects in the wild use Autotools/Cmake/Meson in non-portable ways and hence need to be changed for portability to provide the same seamless experience. The general rule of thumb is: Build systems should focus on building and not do dependency management tasks other than finding library paths (e.g. using CMake’s official find_package functionality). Then, cross-compilation and static compilation come for free most of the time.

Which Cross-Architectures Are Available?

Cross-compilation is generally possible on one of the given axes or a mix of those:

different CPU architectures
different operating systems (/system libraries, including libc)
dynamic or static linking

Their combination can be considered a platform. We can look the available platforms up for our given nixpkgs pin:

$ nix repl

nix-repl> :lf .
nix-repl> pkgs = import inputs.nixpkgs {}
nix-repl> pkgs.pkgsCross.<Press TAB>
pkgs.pkgsCross.aarch64-android             pkgs.pkgsCross.loongarch64-linux           pkgs.pkgsCross.ppc64-musl
pkgs.pkgsCross.aarch64-android-prebuilt    pkgs.pkgsCross.m68k                        pkgs.pkgsCross.ppcle-embedded
pkgs.pkgsCross.aarch64-darwin              pkgs.pkgsCross.mingw32                     pkgs.pkgsCross.raspberryPi
pkgs.pkgsCross.aarch64-embedded            pkgs.pkgsCross.mingwW64                    pkgs.pkgsCross.remarkable1
pkgs.pkgsCross.aarch64-multiplatform       pkgs.pkgsCross.mips-embedded               pkgs.pkgsCross.remarkable2
pkgs.pkgsCross.aarch64-multiplatform-musl  pkgs.pkgsCross.mips-linux-gnu              pkgs.pkgsCross.riscv32
pkgs.pkgsCross.aarch64be-embedded          pkgs.pkgsCross.mips64-embedded             pkgs.pkgsCross.riscv32-embedded
pkgs.pkgsCross.arm-embedded                pkgs.pkgsCross.mips64-linux-gnuabi64       pkgs.pkgsCross.riscv64
pkgs.pkgsCross.armhf-embedded              pkgs.pkgsCross.mips64-linux-gnuabin32      pkgs.pkgsCross.riscv64-embedded
pkgs.pkgsCross.armv7a-android-prebuilt     pkgs.pkgsCross.mips64el-linux-gnuabi64     pkgs.pkgsCross.rx-embedded
pkgs.pkgsCross.armv7l-hf-multiplatform     pkgs.pkgsCross.mips64el-linux-gnuabin32    pkgs.pkgsCross.s390
pkgs.pkgsCross.avr                         pkgs.pkgsCross.mipsel-linux-gnu            pkgs.pkgsCross.s390x
pkgs.pkgsCross.ben-nanonote                pkgs.pkgsCross.mmix                        pkgs.pkgsCross.sheevaplug
pkgs.pkgsCross.bluefield2                  pkgs.pkgsCross.msp430                      pkgs.pkgsCross.ucrt64
pkgs.pkgsCross.fuloongminipc               pkgs.pkgsCross.musl-power                  pkgs.pkgsCross.vc4
pkgs.pkgsCross.ghcjs                       pkgs.pkgsCross.musl32                      pkgs.pkgsCross.wasi32
pkgs.pkgsCross.gnu32                       pkgs.pkgsCross.musl64                      pkgs.pkgsCross.x86_64-darwin
pkgs.pkgsCross.gnu64                       pkgs.pkgsCross.muslpi                      pkgs.pkgsCross.x86_64-embedded
pkgs.pkgsCross.i686-embedded               pkgs.pkgsCross.or1k                        pkgs.pkgsCross.x86_64-freebsd
pkgs.pkgsCross.iphone32                    pkgs.pkgsCross.pogoplug4                   pkgs.pkgsCross.x86_64-netbsd
pkgs.pkgsCross.iphone32-simulator          pkgs.pkgsCross.powernv                     pkgs.pkgsCross.x86_64-netbsd-llvm
pkgs.pkgsCross.iphone64                    pkgs.pkgsCross.ppc-embedded                pkgs.pkgsCross.x86_64-unknown-redox
pkgs.pkgsCross.iphone64-simulator          pkgs.pkgsCross.ppc64

There’s also pkgs.pkgsStatic which automatically builds static binaries that are linked against musl instead of the GNU libc.

Can we cross compile from x86_64-linux to aarch64-darwin or the other way around? Compiling from macOS for Linux just works. Trying it out the other way around throws this error:

error: don't yet have a `targetPackages.darwin.LibsystemCross for x86_64-apple-darwin`

It might be generally possible, but no one implemented that, yet.

Summary

This part of our series has illuminated how to package our C++ application in a manner that makes it easily accessible and usable by end-users, without them needing to understand the intricacies of the build process. Moreover, we’ve seen the simplicity and effectiveness of cross-compiling for different architectures using Nix. The ability to compile for multiple platforms with minimal hassle is a game-changer, especially in an environment where time and resources are precious.

There is no real difference between our own packages and the “real” packages from nixpkgs.
nix build and nix run also work on remote repository URLs without cloning the repo.
If we created a library package instead of an executable package, other build recipes (as our package.nix file) can consume it like any other first-class package
As long as our code, build system, and Nix recipe are portable, cross-compilation basically comes for free via pkgs.pkgsCross.
This is also the way to get statically linked packages.

I hope this demonstrates to you that Nix is not just a tool but a powerful ally in the software development process, providing flexibility, consistency, and efficiency. These qualities make Nix an invaluable asset for companies looking to streamline their development pipeline and maintain high standards across diverse computing environments.