C++ with Nix in 2023, Part 1: Developer Shells
![C++ with Nix in 2023, Part 1: Developer Shells](/_astro/keep-calm-use-nix.DYFC_EVD_Z1SCrEz.webp)
In today’s complex software development landscape, C++ developers struggle with the challenges of setting up reliable, consistent, and portable development environments. Traditional methods often involve wrestling with system dependencies, containers, VMs, or even facing discrepancies across different platforms or team members’ setups. Enter Nix: A game-changer in this space, allowing developers to create build environments that are easy to use and maintain.
Why should you, as a C++ developer, invest time in looking at Nix? Ask yourself or your colleagues the following questions:
- “How long did the setup take on day 1 after joining our team?”
- “How long does it take to update our toolchain and deps?”
- “How many days has it been since someone said works on my machine?
- “How often does it happen that it works locally but not in the CI?”
- “Do we regularly push CI fixes just to wait and see if they make the CI green?”
If the answers for setup/update questions are in the region of multiple hours or even days, Nix is surely a must-see for you - companies that use Nix, typically successfully reduce this time to minutes.
Nix streamlines the process of getting your project up and running across multiple systems, ensuring that everyone works within an identical environment. This not only saves time but significantly reduces the “It works on my machine” type of issues. This article series explores how Nix can revolutionize your C++ development workflow. In part 1, we concentrate on starting with a new little example C++ project and its development environment. The following parts will show how to build packages, cross-compile them, and finally describe a simple CI using GitHub Actions.
Get Toolchain and Dependencies with Nix
Before we begin coding anything, we need to obtain a C++ compiler, a build tool, and dependency libraries.
Instead of installing these system-wide on our system or a VM, or building a Docker image, we create a Nix Development shell that is much handier and also doesn’t touch the rest of our system.
As we’re using Nix in this article, please install it first: The Determinate Systems Nix installer is the currently recommended way to do this:
curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install
Ideally, do not use your native package manager to install Nix. Continue reading after the installation.
The modern way to do this with Nix is by creating a flake.nix
file in the root
folder of the project (this is more of a convention than a technical
requirement) that describes the packages and shell environments of this project
(and much more, but we don’t go that far in this article).
The command nix flake init
creates a vanilla flake, but we are going to use
Nix flakes with the flake-parts library, which has a lot
of helpful functionality that we only touch on the surface in this article.
It will however help us provide the same package for multiple architectures
automatically.
This has nothing to do with cross-compilation just yet, but we will come to that
towards the end of the article.
The flake-parts project provides a template for initializing a new project:
nix flake init -t github:hercules-ci/flake-parts
We change the generated template to look like this:
{
description = "C++ Development with Nix in 2023";
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
};
outputs = inputs@{ flake-parts, ... }:
flake-parts.lib.mkFlake { inherit inputs; } {
# This is the list of architectures that work with this project
systems = [
"x86_64-linux" "aarch64-linux" "aarch64-darwin" "x86_64-darwin"
];
perSystem = { config, self', inputs', pkgs, system, ... }: {
# devShells.default describes the default shell with C++, cmake, boost,
# and catch2
devShells.default = pkgs.mkShell {
packages = with pkgs; [
# C++ Compiler is already part of stdenv
boost
catch2
cmake
];
};
};
};
}
To finally obtain all the deps and toolchain-related packages, run:
nix develop
This works on Intel and ARM based GNU/Linux and
macOS.
(Running nix flake show
displays the expanded list of development shells for
all systems mentioned in the systems
variable, thanks to flake-parts
.)
The first invocation takes some time due to the many downloads. The second one will be very quick.
Inside this shell, we can now run cmake
and c++
or g++
(clang++
is the
default on macOS instead of GCC).
The tools are gone again after leaving the shell.
This way we can have completely different dev shells per project without
cluttering up our system.
Adding new dependencies can be done by adding them to the list in the flake and
then re-running the nix develop
shell (which is by default a normal bash that
you can exit from by entering exit
or pressing ctrl-D).
We can also see that Nix created a flake.lock
file that exactly describes
which commit of the nixpkgs
repo was used.
This way colleagues will get exactly the same checkout - even years later!
So Where Do Compiler and Toolchain Now Come From?
$ which c++
/nix/store/zlzz2z48s7ry0hkl55xiqp5a73b4mzrg-gcc-wrapper-12.3.0/bin/c++
$ which cmake
/nix/store/5h0akwq4cwlc3yp92i84nfgcxpv5xv79-cmake-3.26.4/bin/cmake
The nix develop
command downloaded all needed packages and stored them in
/nix/store/...
.
Each package with executables has its own /nix/store/...packagename/bin/
subfolder where all the executables reside.
The nix develop
command does then prepend these bin/
subfolders to our
$PATH
environment, so they
become available for use.
No matter how old or new the packages are, they don’t interfere with the rest of the system, regardless of the Linux distro (or macOS version). Nix also installed all needed shared library dependencies of the tools:
$ ldd $(which cmake)
linux-vdso.so.1 (0x00007ffff7fc8000)
libdl.so.2 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/libdl.so.2 (0x00007ffff7fbd000)
libcurl.so.4 => /nix/store/bsn22x0c23vl21wj5q8fv4ia0r2qbwhk-curl-8.4.0/lib/libcurl.so.4 (0x00007ffff7f03000)
libexpat.so.1 => /nix/store/4zb4ivz3y2sx8xvjbv1wwqzrsbcp3jai-expat-2.5.0/lib/libexpat.so.1 (0x00007ffff7ed8000)
libarchive.so.13 => /nix/store/zjc21mkk7zabk0w2kp30awdp0ix2v0rk-libarchive-3.7.2-lib/lib/libarchive.so.13 (0x00007ffff7e10000)
librhash.so.1 => /nix/store/xzamy1xhaybmp8587ipkhmr3h42nl6h8-rhash-1.4.4/lib/librhash.so.1 (0x00007ffff7dd9000)
libuv.so.1 => /nix/store/hba5ja1cri7jd7gvw77dqxsvsccijhz8-libuv-1.46.0/lib/libuv.so.1 (0x00007ffff7da6000)
# and many more entries ...
This ldd
call shows that all
library dependencies reference exact package paths in the Nix store.
Each referenced package has its individual hash in the store path name.
That means that we can have many different versions of the same applications
and libraries on the same system and they will never interfere with each other.
But let’s cut this topic off and concentrate on C++ again.
Let’s Start Coding
I prepared the code in this GitHub repository, in the branch part1
:
https://github.com/tfc/cpp-nix-2023/tree/part1
Let’s copy its CMakeLists.txt
file, src/
, and test/
folders into the same
location as our flake.nix
file.
The application is not much more than a Hello World app that depends on the Boost library, but just to print its version. For testing, we use the Catch2 library.
We can now configure, compile, test, and run the code like this:
$ cmake -B build -S .
$ cd build
$ cmake --build .
$ ctest
Test project /home/tfc/src/cpp-nix-2023/build
Start 1: Scenario: basic math
1/1 Test #1: Scenario: basic math ............. Passed 0.00 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 0.00 sec
# run the app
$ ./cpp-nix
Hello World!
Compiler: g++ 12.3.0
Boost: 1.81.0
The app of course still works even after we left the Nix shell.
How Do Cmake and the C++ Compiler Find Boost and Catch2?
To find Boost and Catch2, the CMakeLists.txt
file uses the find_package
functionality of Cmake:
# file: CMakeLists.txt
# ...
find_package(Boost REQUIRED COMPONENTS system)
# ...
# file: test/CMakeLists.txt
find_package(Catch2 REQUIRED)
# ...
This function searches all paths in the
$CMAKE_PREFIX_PATH
environment variable
for library-specific .cmake
files that contain all the information about the
location of headers and libraries, compile and link parameters, etc.
The nix develop
command also created this environment variable for us.
This happens only when cmake
is used, as this package contains certain hooks.
If we were using Meson or
Autotools, the pkg-config
package would perform similar things for us.
We never need to do anything Nix-specific in any of our projects outside our
.nix
files.
All projects are simply kept in portable shape and then can be managed by Nix
without further changes.
In other terms, we are splitting the concerns of build system management and
dependency management over the tools CMake and Nix.
Create a Dev Shell with Clang instead of GCC
We can add more and more dependencies to our project: Just add them to the list. But how can we for example exchange the compiler? Let’s see by switching from GCC to Clang.
The stdenv
variable contains the C++ toolchain already.
On Linux, this is GCC, on macOS it’s Clang.
So let’s switch to the other compiler now by creating another devShell
attribute next to the default
one:
# file: flake.nix
# ...
devShells = {
default = pkgs.mkShell { ... };
# macOS users use pkgs.gccStdenv instead of pkgs.clangStdenv
clang = pkgs.mkShell.override { stdenv = pkgs.clangStdenv; } {
packages = with pkgs; [
boost
catch2
cmake
];
};
};
# ...
After this change, we have the default
shell and another shell called clang
.
It duplicates the lines of the default
shell but overrides the default
compiler selection in the pkgs.mkShell
function.
(In the next articles we will learn how to avoid this duplication which
obviously wouldn’t scale well for big projects. But one step after the other.)
Whenever we run nix develop
or nix build
without further parameters, it
expects a flake in the current directory that either has a devShells.default
or packages.default
output.
Now we can run nix develop
with an argument that selects the new alternative
shell environment:
$ nix develop .#clang
$ c++ --version
clang version 11.1.0
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /nix/store/axnmbgr83qwzwxim51xg8rp171654jdy-clang-11.1.0/bin
The
.#clang
syntax describes the path to the flake before the#
sign (which can also be a URL to a git repo) and the selected flake output after it.
We can also select a newer compiler by using pkgs.clang16Stdenv
, which is the
newest clang in nixpkgs at the time of writing.
How to know which stdenv
s are available?
There is always https://search.nixos.org, but we can also have a look at
what’s available in our currently pinned selection of nixpkgs (as per our
flake.lock
file):
$ nix repl
nix-repl> :lf .
Added 17 variables.
nix-repl> pkgs = import inputs.nixpkgs {}
nix-repl> pkgs.clang<Press TAB>
pkgs.clang pkgs.clang15Stdenv
pkgs.clang-analyzer pkgs.clang16Stdenv
pkgs.clang-manpages pkgs.clang5Stdenv
pkgs.clang-ocl pkgs.clang6Stdenv
pkgs.clang-sierraHack pkgs.clang7Stdenv
pkgs.clang-sierraHack-stdenv pkgs.clang8Stdenv
pkgs.clang-tools pkgs.clang9Stdenv
pkgs.clang-tools_10 pkgs.clangMultiStdenv
...
Each nixpkgs commit contains multiple versions of GCC and clang.
On the master
or nixos-unstable
branch, the versions are usually newer than
on the nixos-XX.YY
release branches (the latest is nixos-23.05
at the time
of this writing).
The default compiler selection of gcc
or clang
does not always use the
newest version because the default compiler is also used to compile all the
packages.
Therefore the default compiler is switched with care as this can break many
packages which would need to be fixed at the same time (The nixpkgs maintainers
thankfully do this on a regular base).
However, we can still always select the one we like for our packages.
Update the Toolchain
To update the toolchain and libraries, we can now run:
$ nix flake update --commit-lock-file
This will re-download the latest flake inputs, calculate their hashes, and save
them into the flake.lock
file.
The --commit-lock-file
argument is optional but useful as it automatically
creates a commit with a descriptive commit message that lists all changes.
If we realize that the upgrade is breaking, we can fix our code and commit these changes together with the lock file update. This way we can make sure that all commits always work.
Upgrading toolchains and dependencies with Nix generally doesn’t create more hassle than that. The CI doesn’t care what it runs: If it builds a five-year-old commit or a new one with the latest and greatest packages inside, it just works. (We will look at defining CI environments in one of the follow-up articles.)
Dive Deeper into Nix
We’ve touched upon several Nix concepts in this article, but there’s so much
more to this rich ecosystem.
If you felt we skimmed over some aspects, you’re right.
Concepts such as flake-parts, mkShell
, override
,
etc. have much more depth than what’s been presented here.
If you’re keen on becoming proficient and independent with Nix, consider enrolling in the Nixcademy class. From basics to advanced techniques, Nixcademy provides a comprehensive learning experience. For organizations evaluating the integration of Nix into their workflow, Nixcademy offers Proof-of-Concept work, as well as long-term mentoring and consulting. With the right guidance, you can harness the power of Nix to streamline your development processes and mitigate typical software setup issues.
Summary
In this article, we learned how to:
- Create a reproducible environment for developing a C++ package
- Reuse the dev environment specification with different inputs
- Update nixpkgs effortlessly
Here’s what we gathered from our exploration:
Reproducible Environments
: Nix allows C++ developers to create and share a consistent environment for coding. No more discrepancies in libraries or tools across different systems.
No Need for Containers or VMs
: This means fast setup times and less overhead. No more two worlds that are “inside and outside” the build container/VM.
Cross-Platform
: The Nix setup works seamlessly across all Linux distributions and macOS. It ensures uniformity and reduces the challenges of cross-platform development.
Flexibility with Compilers
: With Nix, switching between compilers or even testing your codebase against multiple compilers becomes a breeze. This promotes code that is standard-compliant and truly portable.
Easy Upgrades
: Keeping toolchains and dependencies updated is straightforward with Nix, ensuring that your projects are always running with the latest and most secure versions.
Embracing Nix could be the breakthrough you’ve been looking for in streamlining your C++ development processes. This is just the beginning of our journey with Nix, in the next articles we are going to look into:
- Packaging our app
- Cross-compilation
- Setting up a free GitHub CI for Linux and macOS
Stay tuned and keep innovating!
Continue with part 2 of this article series, where we introduce packaging and cross-compilation!