Unveiling the Power of the NixOS Integration Test Driver (Part 1)

In DevOps and software testing, efficiency and reliability are important. Developers and system administrators are in need of tools and practices that optimize their workflows, ensure the stability of their systems, and save them time and resources. Enter the NixOS Integration Test Driver—the killer feature within the NixOS ecosystem that combines the best of declarative configuration, virtualization, and Python scripting to revolutionize the way we test and validate our systems.

In this two-part series, we will embark on a journey to explore the incredible capabilities of the NixOS Integration Test Driver. In this first article, we have a quick glance of what this powerful tool can do. From orchestrating multiple virtual machines (VMs) to seamlessly running GUI applications with text recognition, we’ll look at real-world scenarios that demonstrate the driver’s versatility and simplicity.

So, whether you’re a seasoned NixOS enthusiast or new to this innovative operating system, fasten your seatbelts as we dive into a world where integration testing becomes an effortless and integral part of your development and deployment process.

The NixOS Integration Test Driver

The nixpkgs project started integrating automatic system tests that run within the nix sandbox in 2009 (the first commit of the integration test driver framework).

From the start, this framework was able to run multiple virtual machines in virtual networks. The developer of each test scenario describes the NixOS configurations of the machines and a script that describes the testing workflow:

The image mentions Python as the implementation language of the driver, which substituted the Perl implementation in 2019. (NixOS 20.03 release notes) There will be more information on that in the next article.

As of today, there are ~300 tests in the nixpkgs repository of which a large sub-set of tests is run on every NixOS release.

The minimal implementation of a test with two machines in a virtual network that ping each other would look like this:

pkgs.testers.runNixOSTest {
  name = "An awesome test.";

  nodes = {
    machine1 = { pkgs, ... }: { };
    machine2 = { pkgs, ... }: { };
  };

  testScript = ''
    machine1.wait_for_unit("network-online.target")
    machine2.wait_for_unit("network-online.target")

    machine1.succeed("ping -c 1 machine2")
    machine2.succeed("ping -c 1 machine1")
  '';
}

This test creates the machines machine1 and machine2, which contain empty NixOS configurations. This means that the NixOS integration test driver creates VMs with a default setup that run as qemu VMs later.

The testScript part contains the actual test: Both machines are available as Python objects that provide many methods for controlling the machines and running commands. The variable names and host names in the virtual network are identical to the machine names in the nodes attribute set.

What this test does is simple:

1. Wait for both machines to boot until they are online (by waiting for the systemd network-online.target)
2. Let them ping each other.

This specific example test is ready to run on github. Run it on your machine right now with this command (Which assumes that you have a flakes-enabled Nix installation and KVM):

nix -L build github:tfc/nixos-integration-test-example

This command works if you have a flake-enabled Nix installation on your KVM-enabled Linux machine. The test can potentially work on macOS, but more about that in the next article. If this command takes more than a minute on a machine that is less than 5 years old, please check if the nixbld group has access to /dev/kvm.

On my laptop, the whole test takes ~30 seconds (excluding the time needed to download all dependencies when running it for the first time, of course!). After changing the configuration of one of the machines, rebuilding the VMs and the test also only takes ~40 seconds. How can it rebuild a whole VM in roughly 10 seconds? We will learn about this in the next article.

If you are eager to play with the test driver before we go in depth in the next article, have a look at the NixOS integration test driver documentation. The hundreds of existing tests in the nixpkgs repository are also worth having a look at to inspire yourself.

Examples of Impressive Integration Tests

Let’s have a look at some of the impressive existing tests in nixpkgs/nixos/tests to highlight the potential and practicality of this mighty test framework.

BitTorrent Testing with private NAT networks

The nixpkgs/nixos/tests/bittorrent.nix test describes 4 different hosts in 2 different networks like in this image:

What the test does is the following:

1. Create a file that shall be shared and register it on the tracker’s Opentracker daemon.
2. Let client1 open the file share link with the Transmission app and wait until the download completes. Verify the downloaded file.
3. Switch off trackers’s instance of Transmission, so client1 is the only peer that has this file.
4. Let client2 open the file share link to get the file. Note that it must be obtained via client1 who is behind a NAT router with a firewall. This scenario implicitly tests if the app can successfully open a hole in the firewall via UPNP.

The declarative part of the test that defines the infrastructure is only about 50 Lines of Code long.

On my laptop, it takes 1 minute and 15 seconds to run (excluding dependency downloads). How to run it:

git clone https://github.com/nixos/nixpkgs --depth=1
cd nixpkgs
nix-build -A nixosTests.bittorrent

Chrome and Chromium Sandbox Test

The Google Chrome test verifies that the browser runs with the activated sandboxing feature. What’s interesting about this is that there doesn’t seem to exist any command line parameter to Chrome that tells about the status of this feature on the terminal.

To work around this, the nixpkgs/nixos/tests/chromium.nix test runs a VM with full graphical desktop support to run Chrome inside that. Then, it opens the page about://sandbox and copies its content into the system paste-buffer. The content of the buffer is then checked against the substring “You are adequately sandboxed” among others.

Taking this screenshot where this happens is part of the test:

This test is also defined for different variants of Google Chrome and Chromium packages, by making the selected package a parameter to the test function on nix expression level.

On my laptop, it takes 1 minute and 40 seconds to run (excluding dependency downloads). How to run it:

git clone https://github.com/nixos/nixpkgs --depth=1
cd nixpkgs
nix-build -A nixosTests.chromium

Gnome Desktop Environment Test

The other screenshots showed a very minimal desktop environment around the tested apps. But this does not mean that we can’t have tests with some heavier desktop environments: nixpkgs/nixos/tests/gnome.nix tests the GNOME desktop on NixOS.

Multiplayer Game Tests

I didn’t want to withhold the tests of the multiplayer games Openarena in nixos/tests/openarena.nix and Teeworlds in nixos/tests/teeworlds.nix from you:

Both tests run a game server instance and two clients. In the end, both clients must be able to successfully connect to the server and be visible to the respective other client.

All the screenshots in this article are taken automatically by their respective tests.

VirtualBox Tests

The nixpkgs/nixos/tests/virtualbox.nix tests check the functionality of various aspects of VirtualBox.

In 500 lines of code, the test creates multiple minimal guest VMs and runs them in Virtualbox, with and without GUI, to test different functions of VirtualBox:

MySQL Replication Test

The nixos/nixpkgs/tests/mysql/mysql-replication.nix is also noteworthy, as it does:

1. Start a primary MySQL instance
2. Start two secondary ones and wait until they finish synchronizing with the primary.
3. Simulate downtime by stopping one of the secondary instances.
4. Feed the primary with more data.
5. Restart the secondary instance again and wait until it finishes re-synchronizing with the primary again.

Summary

In the first part of our short series, we’ve uncovered the incredible capabilities of the NixOS Integration Test Driver, a powerful tool that simplifies and supercharges integration testing. Here’s a glimpse of what we’ve learned:

Minimal Requirements: NixOS integration tests have minimal prerequisites, making them accessible to a wide range of developers. With just a Linux system and KVM support, you can run complex tests without the need for additional infrastructure.

Effortless Debugging: When an integration test fails in your continuous integration (CI) pipeline, the NixOS Integration Test Driver allows you to run these tests locally for debugging. This streamlined process can save developers valuable time and frustration.

Speed and Reliability: The framework’s speed and reliability are unparalleled. It facilitates the creation of robust tests that don’t flake, ensuring consistent and dependable results.

Realistic Simulations: Particularly when NixOS is your deployment platform of choice, these tests can closely simulate actual customer deployments. This realism enhances your ability to catch issues before they reach production.

In the second article of this series, we dive deeper into the NixOS Integration Test Driver. We’ll explore the inner workings of the driver’s architecture and provide insights on how to run it effectively. You’ll discover the interactive mode, a powerful feature that simplifies local debugging and exploration, along with a look at linting and type checking of integration tests.

As a bonus, remember that Nixcademy is here to assist you on your journey with the NixOS Integration Test Driver. Our services include staff training to help your team harness the full potential of this tool, evaluation to determine how it fits into your organization’s testing and deployment strategy, and even building proof-of-concept projects to demonstrate its value in action. Stay tuned for Article 2, and let’s continue to explore the world of NixOS integration testing together.