Build and Deploy Linux Systems from macOS
![Build and Deploy Linux Systems from macOS](/_astro/penguin-jumps-from-apple.Bs-hkf9u_1jp2wx.webp)
Macbooks stand out with their long battery life and the “just works” experience with meeting multimedia. Developing for Linux and deploying to Linux machines with Nix, however, was a pretty much reduced experience until lately. In this article we’re going to look closer at how easy it is to configure a macOS system with a “local remote builder” so it can build NixOS images and also remote deploy to Linux systems without the need for external remote builders.
It is generally possible to cross-compile between different architectures, but not all software packages can be cross-compiled. So sometimes, we need a Linux machine to delegate Linux builds to. This article is about running a NixOS VM in the background on your macOS system.
The story unfolds in chronological order:
At first, there was the pkgs.darwin.linux-builder
package which worked well
from the beginning, but was laborious to set up.
Then, a nix-darwin module was created that made the setup a literal one-liner.
Let’s see how that works and how to improve it.
🏗 The Linux Builder in nixpkgs
In December 2022, Gabriella Gonzalez
upstreamed her streamlined version of a Linux builder
that runs on macOS and
blogged about it.
It consists of a
NixOS profile module that sets up a minimal qemu NixOS VM
along with a ready-to-use SSH key pair.
As the official NixOS documentation of the darwin.linux-builder
package states,
it just needs to be run and can then be set up as a remote builder.
As this VM is a Linux image, we would technically first have to find a way to build a Linux system on macOS, which is what we wanted from the beginning. As it is upstream now, it can be obtained from the official NixOS binary cache without having to bootstrap it first.
To run this VM, we do not need to install Qemu or any other virtualization solution first, as this package itself is a script that runs Qemu for us:
nix run nixpkgs#darwin.linux-builder
Our local Nix daemon still needs to be educated about the availability of this runner. To achieve that, we need to perform the following changes to our system:
- Add ourselves to the
extra-trusted-users
variable in the nix settings - Add the builder to the
builders
variable in the nix settings - Add a new SSH config file that maps the hostname
linux-builder
to localhost but with the builder’s port - Restart the nix daemon
Unfortunately, this way we have to take care of the process and its setup ourselves. This seems laborious and is surely not the declarative workflow that we’re used to from Nix(OS) otherwise already.
▶️ The nix-darwin Option
Don’t worry about the foregoing steps if you are a nix-darwin user, because in July 2023, NixOS contributor Enzime upstreamed a nix-darwin module that makes enabling the linux-builder VM and doing all the setup a declarative one-liner.
Please have a look at our last macOS-related article if you have not heard about nix-darwin, yet.
This is the relevant nix-darwin configuration snippet that installs and sets up the linux-builder VM on our system:
# file: nix-darwin configuration.nix
{
nix = {
linux-builder.enable = true;
# This line is a prerequisite
settings.trusted-users = [ "@admin" ];
};
}
With this snippet in your nix-darwin configuration, run the
darwin-rebuild switch
command to activate the new builder.
(If you forgot how to run this command or which parameters you need, please have
another look at the article where we introduced nix-darwin).
As already briefly mentioned in the last article, it adds the following things to our system setup:
- A new launchd
service called
org.nixos.linux-builder
that keeps SSH keys and VM disk image in/var/lib/darwin-builder
(Runsudo launchctl list org.nixos.linux-builder
to inspect it) - A new file
/etc/ssh/ssh_config.d/100-linux-builder.conf
that creates an SSH host-aliaslinux-builder
(this one is needed for TCP port bending reasons) /etc/nix/machines
gets the needed remote builder entry
The builder can be tested like this:
$ nix build \
--impure \
--expr '(with import <nixpkgs> { system = "aarch64-linux"; }; runCommand "foo" {} "uname -a > $out")'
$ cat result
Linux localhost 6.1.72 #1-NixOS SMP Wed Feb 12 16:10:37 UTC 2024 aarch64 GNU/Linux
This example assumes a Mac with Apple silicon. The builder works on Intel Macs, too, but then you need to change the
system
variable tox86_64-linux
.
The builder is relatively slow when building real-life packages due to its default configuration.
⤴️ Improving the Linux Builder Setup
The good thing about having a Linux builder running as a service in the background is that it enables us to build a new Linux builder with a slightly improved configuration.
To make the builder faster by giving it more CPUs, RAM, and a bigger disk image (the defaults are 1 CPU core, 3GB RAM, and 20GB disk), we can simply extend our nix-darwin configuration:
{
nix.linux-builder = {
enable = true;
ephemeral = true;
maxJobs = 4;
config = {
virtualisation = {
darwin-builder = {
diskSize = 40 * 1024;
memorySize = 8 * 1024;
};
cores = 6;
};
};
};
}
In this example, we give the Linux builder 6 CPU cores, 8GB RAM, and 40GB disk
size.
The maxJobs
variable sets the
nix.buildMachines.linux-builder.maxJobs
config attribute that defines how many jobs may be delegated to this builder
concurrently.
After the Linux builder’s configuration has changed, it needs to be stopped to have its disk image deleted before it is restarted again. The reason is that some configuration changes don’t play well with the state of the disk image (especially disk image changes!).
To get rid of this additional manual step,
I upstreamed the
nix.linux-builder.ephemeral
flag in January 2024.
It controls whether the disk image of the builder VM shall be deleted on every
restart.
It can be set whenever the builder’s configuration is changed and deleted
afterward.
I tend to keep it enabled as I don’t like my builders to pile up state.
With this changed builder configuration in your nix-darwin configuration, run
the darwin-rebuild switch
command again to activate the improved builder.
(If you forgot how to run this command or which parameters you need, please have
another look at the article where we introduced nix-darwin).
Testing Remote Deployments to Linux Machines
Apart from the little test earlier, we can check that we can now deploy a NixOS configuration to a remote NixOS Linux machine.
To do so, we need a complete NixOS configuration of some NixOS machine in a
repository (not included in this article).
In this example, I deploy to an aarch64-linux
system with the hostname
build02
from a configuration repository that I already had available.
$ nixos-rebuild switch \
--fast \
--target-host build02 \
--flake .#build02 \
--use-remote-sudo \
--use-substitutes
building the system configuration...
copying 836 paths...
copying path '/nix/store/...' to 'ssh://build02'...
# ....
Shared connection to build02 closed.
stopping the following units: ...
NOT restarting the following changed units: ...
activating the configuration...
setting up /etc...
restarting systemd...
reloading user units for tfc...
setting up tmpfiles
reloading the following units: ...
restarting the following units: ...
starting the following units: ...
Shared connection to build02 closed.
As nixos-rebuild
is not pre-installed on macOS, you can either install
it via your nix-darwin config or launch a nix shell:
nix shell nixpkgs#nixos-rebuild
If the nixos-rebuild
command or flags for remote deploying are not familiar
to you, please have a look at
part 1
and
part 2
of our nixos-rebuild
articles.
To be honest, it still took longer than on Linux, but it just works.
If you try this at home and run into
this error, then update your
nixos-rebuild
to the latest one from master, where I recently upstreamed
another fix.
Conclusion
In wrapping up, the journey from needing a better way to work on/deploy Linux projects from macOS to having a reliable solution in place shows the power of community-driven development.
With the detailed steps we’ve gone through, from basic setup to enhancing its capabilities, this tool is now accessible to more macOS users without much hassle. And with the introduction of an “ephemeral” feature flag, it’s even easier to manage and maintain. Thanks to the efforts of Gabriella Gonzalez, Enzime, and the broader Nix community, the gap between macOS and Linux development is now easier to bridge.
However, it still does not bridge the gap between CPU architectures: It would be great to be able to use the Rosetta feature in the NixOS VM, as there is a Rosetta flag for NixOS guest systems, which is unfortunately not supported by Qemu. There is ongoing work in that area - stay tuned!