Build and Deploy Linux Systems from macOS

Macbooks stand out with their long battery life and the “just works” experience with meeting multimedia. Developing for Linux and deploying to Linux machines with Nix, however, was a pretty much reduced experience until lately. In this article we’re going to look closer at how easy it is to configure a macOS system with a “local remote builder” so it can build NixOS images and also remote deploy to Linux systems without the need for external remote builders.

It is generally possible to cross-compile between different architectures, but not all software packages can be cross-compiled. So sometimes, we need a Linux machine to delegate Linux builds to. This article is about running a NixOS VM in the background on your macOS system.

The story unfolds in chronological order: At first, there was the pkgs.darwin.linux-builder package which worked well from the beginning, but was laborious to set up. Then, a nix-darwin module was created that made the setup a literal one-liner. Let’s see how that works and how to improve it.

🏗 The Linux Builder in nixpkgs

In December 2022, Gabriella Gonzalez upstreamed her streamlined version of a Linux builder that runs on macOS and blogged about it. It consists of a NixOS profile module that sets up a minimal qemu NixOS VM along with a ready-to-use SSH key pair. As the official NixOS documentation of the darwin.linux-builder package states, it just needs to be run and can then be set up as a remote builder.

As this VM is a Linux image, we would technically first have to find a way to build a Linux system on macOS, which is what we wanted from the beginning. As it is upstream now, it can be obtained from the official NixOS binary cache without having to bootstrap it first.

To run this VM, we do not need to install Qemu or any other virtualization solution first, as this package itself is a script that runs Qemu for us:

nix run nixpkgs#darwin.linux-builder

Our local Nix daemon still needs to be educated about the availability of this runner. To achieve that, we need to perform the following changes to our system:

1. Add ourselves to the extra-trusted-users variable in the nix settings
2. Add the builder to the builders variable in the nix settings
3. Add a new SSH config file that maps the hostname linux-builder to localhost but with the builder’s port
4. Restart the nix daemon

Unfortunately, this way we have to take care of the process and its setup ourselves. This seems laborious and is surely not the declarative workflow that we’re used to from Nix(OS) otherwise already.

▶️ The nix-darwin Option

Don’t worry about the foregoing steps if you are a nix-darwin user, because in July 2023, NixOS contributor Enzime upstreamed a nix-darwin module that makes enabling the linux-builder VM and doing all the setup a declarative one-liner.

Please have a look at our last macOS-related article if you have not heard about nix-darwin, yet.

This is the relevant nix-darwin configuration snippet that installs and sets up the linux-builder VM on our system:

# file: nix-darwin configuration.nix
{
  nix = {
    linux-builder.enable = true;

    # This line is a prerequisite
    settings.trusted-users = [ "@admin" ];
  };
}

With this snippet in your nix-darwin configuration, run the darwin-rebuild switch command to activate the new builder. (If you forgot how to run this command or which parameters you need, please have another look at the article where we introduced nix-darwin).

As already briefly mentioned in the last article, it adds the following things to our system setup:

• A new launchd service called org.nixos.linux-builder that keeps SSH keys and VM disk image in /var/lib/darwin-builder (Run sudo launchctl list org.nixos.linux-builder to inspect it)
• A new file /etc/ssh/ssh_config.d/100-linux-builder.conf that creates an SSH host-alias linux-builder (this one is needed for TCP port bending reasons)
• /etc/nix/machines gets the needed remote builder entry

The builder can be tested like this:

$ nix build \
  --impure \
  --expr '(with import <nixpkgs> { system = "aarch64-linux"; }; runCommand "foo" {} "uname -a > $out")'
$ cat result
Linux localhost 6.1.72 #1-NixOS SMP Wed Feb 12 16:10:37 UTC 2024 aarch64 GNU/Linux

This example assumes a Mac with Apple silicon. The builder works on Intel Macs, too, but then you need to change the system variable to x86_64-linux.

The builder is relatively slow when building real-life packages due to its default configuration.

⤴️ Improving the Linux Builder Setup

The good thing about having a Linux builder running as a service in the background is that it enables us to build a new Linux builder with a slightly improved configuration.

To make the builder faster by giving it more CPUs, RAM, and a bigger disk image (the defaults are 1 CPU core, 3GB RAM, and 20GB disk), we can simply extend our nix-darwin configuration:

{
  nix.linux-builder = {
    enable = true;
    ephemeral = true;
    maxJobs = 4;
    config = {
      virtualisation = {
        darwin-builder = {
          diskSize = 40 * 1024;
          memorySize = 8 * 1024;
        };
        cores = 6;
      };
    };
  };
}

In this example, we give the Linux builder 6 CPU cores, 8GB RAM, and 40GB disk size. The maxJobs variable sets the nix.buildMachines.linux-builder.maxJobs config attribute that defines how many jobs may be delegated to this builder concurrently.

After the Linux builder’s configuration has changed, it needs to be stopped to have its disk image deleted before it is restarted again. The reason is that some configuration changes don’t play well with the state of the disk image (especially disk image changes!).

To get rid of this additional manual step, I upstreamed the nix.linux-builder.ephemeral flag in January 2024. It controls whether the disk image of the builder VM shall be deleted on every restart. It can be set whenever the builder’s configuration is changed and deleted afterward. I tend to keep it enabled as I don’t like my builders to pile up state.

With this changed builder configuration in your nix-darwin configuration, run the darwin-rebuild switch command again to activate the improved builder. (If you forgot how to run this command or which parameters you need, please have another look at the article where we introduced nix-darwin).

Testing Remote Deployments to Linux Machines

Apart from the little test earlier, we can check that we can now deploy a NixOS configuration to a remote NixOS Linux machine.

To do so, we need a complete NixOS configuration of some NixOS machine in a repository (not included in this article). In this example, I deploy to an aarch64-linux system with the hostname build02 from a configuration repository that I already had available.

$ nixos-rebuild switch \
    --fast \
    --target-host build02 \
    --flake .#build02 \
    --use-remote-sudo \
    --use-substitutes
building the system configuration...
copying 836 paths...
copying path '/nix/store/...' to 'ssh://build02'...
# ....
Shared connection to build02 closed.
stopping the following units: ...
NOT restarting the following changed units: ...
activating the configuration...
setting up /etc...
restarting systemd...
reloading user units for tfc...
setting up tmpfiles
reloading the following units: ...
restarting the following units: ...
starting the following units: ...
Shared connection to build02 closed.

If the nixos-rebuild command or flags for remote deploying are not familiar to you, please have a look at part 1 and part 2 of our nixos-rebuild articles.

To be honest, it still took longer than on Linux, but it just works.

Conclusion

In wrapping up, the journey from needing a better way to work on/deploy Linux projects from macOS to having a reliable solution in place shows the power of community-driven development.

With the detailed steps we’ve gone through, from basic setup to enhancing its capabilities, this tool is now accessible to more macOS users without much hassle. And with the introduction of an “ephemeral” feature flag, it’s even easier to manage and maintain. Thanks to the efforts of Gabriella Gonzalez, Enzime, and the broader Nix community, the gap between macOS and Linux development is now easier to bridge.

However, it still does not bridge the gap between CPU architectures: It would be great to be able to use the Rosetta feature in the NixOS VM, as there is a Rosetta flag for NixOS guest systems, which is unfortunately not supported by Qemu. There is ongoing work in that area - stay tuned!