Nix IFD: A Ticking Time Bomb in Your Build Pipeline?
In the world of Nix, there’s a controversial technique that can either turbocharge your development or slam the brakes on your build times: Import From Derivation (IFD). IFD makes managing dependencies easier, but it can also make things complicated.
This article explains how IFD works and why it can be good or bad. It helps you understand when to use IFD and when to avoid it. Read on to make your Nix builds run smoothly!
What is IFD?
IFD, Import From Derivation, is a technique that is used for building packages with e.g. automatic dependency resolution. To understand how IFD works, let’s first have a look at how non-IFD builds work:
The Phases of Nix Builds
When we build a package, several things happen, but this is not immediately visible to us.
This command builds the GNU hello app with a single command from the latest nixpkgs checkout:
$ nix build --print-out-paths nixpkgs#hello
/nix/store/9l3rahv16sipckaksiw988lxyfcihql9-hello-2.12.1
$ /nix/store/9l3rahv16sipckaksiw988lxyfcihql9-hello-2.12.1/bin/hello
Hello, world!
(The --print-out-paths flag is generally not needed but for this article, it is useful because it prints the output path that contains the package.)
When we run this command, the package build happens in multiple phases:
- Fetch the latest
nixpkgscheckout that contains the nix expression behindnixpkgs#hello - Evaluate the nix expression and create a derivation
- Realize the derivation and create an output path
Eelco’s original PhD thesis about Nix illustrates these phases like this:
The older, pre-flakes commands nix-instantiate and nix-store allow us to perform a nix build in the individual steps:
$ nix-instantiate ~/src/nixpkgs -A pkgs.hello
/nix/store/dwkzl7flwsi6rjginyhv9driwvyqf1s4-hello-2.12.1.drv
$ nix-store --realise /nix/store/dwkzl7flwsi6rjginyhv9driwvyqf1s4-hello-2.12.1.drv
/nix/store/99hmg51vhxrapv3n0d7afr10mlsyyq68-hello-2.12.1
$ /nix/store/99hmg51vhxrapv3n0d7afr10mlsyyq68-hello-2.12.1/bin/hello
Hello, world!
With this sequence of commands, it’s obvious that the derivation step creates a .drv file in the nix store.
When you open this file with a text editor, you will recognize that it contains structured data that refers to other nix store paths, but it essentially does not contain any nix expressions.
The realization step creates the package from the structured information in the .drv file.
.drv files can also easily be transferred to other hosts for parallel distributed compilation.
The following diagram illustrates what happens in a time sequence and what we are waiting for in each step:
Of course, the evaluation is typically rather quick, and what we’re waiting for most of the time is the package build itself.
What is interesting, though, is that the evaluation results in just one .drv path that is printed on the terminal.
However, this .drv file refers to potentially many other .drv files.
These are its dependencies, as most packages are not standalone but need libraries to run etc.
What the nix daemon sees after evaluation is a graph of derivations that depend on each other.
With this information, the nix daemon regards each .drv file as a work package and parallelizes them as much as possible:
In this example, package D might be our hello package, and the other derivations are its dependencies, like the C library, etc. The daemon will try to download as much as possible from binary caches and only rebuild what is not available in the cache.
What is Import From Derivation (IFD)?
The output of a realized derivation is mostly executable binaries, libraries, etc. but they don’t have to. A realized derivation can also result in a folder with PDFs, images, XML/JSON documents, etc. - but also nix expressions that might have been generated from some other input. This happens quite often (see also the Practical Examples of IFD section later) and the consequence is that the Nix evaluator has to wait for the Nix builder to finish generating nix expressions before it can evaluate them:
In this diagram, Nix expression 1 is the one that imports Nix expression 2. But Nix expression 2 first has to be generated from some input. This leads to two Nix realisation steps.
This time sequence illustrates the user’s experience:
The nix tooling switches hence and forth between evaluation and realisation (building). The current implementation of the logging of these phases is lacking a bit, so for the user, it looks like a long-running evaluation.
The biggest problem potential comes with the fact that evaluation is inherently sequential (at least in the current implementation), while only realization is parallel. This means that builds with multiple cases of IFD do not scale well because they happen strictly one after the other.
Enabling or Disabling IFD
To check if IFD is enabled on your system, run this command:
$ nix config show | grep allow-import-from-derivation
allow-import-from-derivation = true
This setting can be set permanently in the nix configuration file in /etc/nix/nix.conf/ or ~/.config/nix/nix.conf.
How do I see if a project uses IFD?
The nix documentation page on IFD is very clear on that:
Passing an expression
exprthat evaluates to a store path to any built-in function that reads from the filesystem constitutes Import From Derivation (IFD)
Built-in functions that read from the filesystem are for example builtins.import or builtins.readFile, etc.
This explanation is short and clear but it does not help identify if a project uses IFD just by looking at the code, because such built-in functions are used everywhere. However, it might help to look at a concrete example to thoroughly understand what’s going on.
An IFD Example
Let’s get a feeling for how IFD works and what it looks like in action by building a small example that simulates aspects from real life:
# file: ifd.nix
let
pkgs = import <nixpkgs> {};
ifd = pkgs.runCommand "generate-nix-expr" {} ''
mkdir $out
echo "Generating Nix expression..."
sleep 5
cat <<EOF > $out/generated-nix-expr.nix
{ stdenv, lib, hello }:
stdenv.mkDerivation {
name = "hello-ifd";
src = hello.src;
env = lib.optionalAttrs stdenv.hostPlatform.isDarwin {
NIX_LDFLAGS = "-liconv";
};
}
EOF
'';
in
pkgs.callPackage (ifd + "/generated-nix-expr.nix") {}
# |<----------------------------->|
# This expression depends on a
# store path that needs to be
# realized
The generated variable holds a Nix derivation that creates a little hello-ifd package expression.
This package expression is stored under $out/generated-nix-expr.nix with the following content:
{ stdenv, lib, hello }:
stdenv.mkDerivation {
name = "hello-ifd";
src = hello.src;
env = lib.optionalAttrs stdenv.hostPlatform.isDarwin {
NIX_LDFLAGS = "-liconv";
};
}
This example package consumes the source code of the official hello package and rebuilds it as a new package.
The env part is only necessary on macOS systems and can be left out on Linux systems.
In the last line of ifd.nix, we use callPackage on the generated package expression, which is where the import from derivation happens.
The sleep 5 line simulates that generating the Nix package function expression consumes some compute time, as real-life IFD does, although we exagerrate it a bit for demonstration purposes.
(We also could have generated a nix expression like pkgs: pkgs.hello for example, but for this article, I wanted something that has to rebuild to make the full effect visible.)
Let’s run it:
$ nix-build ifd.nix
building '/nix/store/y17ycv7mwm42i0v1j7pjk82l6zjrk2va-generate-nix-expr.drv'...
Generating Nix expression...
this derivation will be built:
/nix/store/4gdds17zwy27mykfqmzq1ad55y6lh3wc-hello-ifd.drv
building '/nix/store/4gdds17zwy27mykfqmzq1ad55y6lh3wc-hello-ifd.drv'...
# ... many lines of build output ...
/nix/store/04hjlmnz5z36c9nlzarhkkwc68hi8syk-hello-ifd
$ /nix/store/04hjlmnz5z36c9nlzarhkkwc68hi8syk-hello-ifd/bin/hello
Hello, world!
We can see that generating the first Nix expression takes 5 seconds. After that, the actual package build runs and provides us with a working Hello-World package.
Let’s change the sleep 5 line to sleep 6 and build it again:
$ nix-build ifd.nix
building '/nix/store/0qg68h7a6wlymmmdlbdvg3q78l1q164l-generate-nix-expr.drv'...
Generating Nix expression...
/nix/store/04hjlmnz5z36c9nlzarhkkwc68hi8syk-hello-ifd
What happens is that the hello-ifd package is not rebuilt because its derivation is the same as before - but we had to wait 6 seconds to find that out.
If we run the same command again, the next invocation will be very quick because neither the generating derivation nor the generated derivation changed.
(This seems totally fine as long as the inputs to the IFD don’t change - but imagine a project where your colleagues don’t really know how to do proper source filtering…)
Practical Examples of IFD
Typically for some language stacks like Python (poetry2nix), Ruby (bundix), Rust (Crane), Javascript/Typescript (node2nix, yarn2nix), Haskell (cabal2nix, haskell.nix), OCaml (opam2nix), etc., projects often come with complex lock files that specify whole lists of package versions that need to be used in the exact combination together to build the working application. There is even dream2nix which aims to be an “automagic” IFD tool for multiple programming language stacks.
One popular example is the crane Nix library for Cargo Rust projects. Let’s look at an example from the Crane documentation:
# ...
myPackage = craneLib.buildPackage {
src = craneLib.cleanCargoSource ./.;
};
With this single call, Crane automatically figures out all the Rust dependencies by their exact version, download hashes, and even the package name from the Rust project. To do this, it parses the Cargo and lock files to create Nix expressions in another sandboxed Nix build, which are then evaluated from IFD context.
Trade-offs and Considerations
In summary, while Import From Derivation offers increased flexibility and dynamic capabilities in Nix builds, it also introduces performance overhead, complexity, and potential pitfalls that should be carefully weighed when considering its use.
Advantages of IFD - Simplified Dependency Management
Without IFD, developers have to generate Nix expressions from e.g. lock files and commit them together with the rest of the code. The disadvantage of this manual approach is that it is easy to forget to keep the lock file and Nix files in sync whenever dependencies are updated. It might also be necessary to exclude the generated Nix files from linter and formatting tools as they are used for example in pre-commit hooks. Also, the generated Nix files might produce huge diffs in the version control system and become annoying noise in pull requests.
With IFD, the lock file becomes the single source of truth because all the Nix expressions are generated on the fly! This way there is no need for manual updates to Nix expressions when dependencies in the lock file change, hence developers can’t forget it.
Not all Nix CI systems support IFD, but Hercules CI for example does.
Disadvantages of IFD
IFD can introduce significant delays during the evaluation phase. Since Nix evaluation is typically single-threaded, incorporating IFD may cause evaluation to stall, leading to longer build times.
However, often developers end up with IFD across multiple projects.
What they then often don’t realize (yet) is that it can become a problem for downstream consumers:
Other developers in the company for example might end up waiting for many IFD evaluations when building system images that consist of many IFD-using packages.
Wait times that may be negligible in an upstream project do then pile up badly in downstream projects - consider this when designing guidelines for your organization!
This is exactly the reason why IFD is not allowed inside the nixpkgs project.
Increased Complexity: The interleaving of evaluation and realization phases inherent in IFD can complicate the build process, making it harder to understand and debug.
Potential for Non-Determinism: If not carefully managed, IFD can introduce non-deterministic behavior into builds, undermining one of Nix’s core strengths.
Increased Complexity of Cross-Platform Builds: In cross-platform builds, IFD can introduce complexity when the derivation being imported needs to execute on a platform different from the one hosting the evaluation. This might impose a burden on the configuration requirements of the developer and build infrastructure.
Conclusion
IFD is an awesome mechanism that can make developer life easier but it is not a silver bullet: Parallel evaluation might become a thing in the future, but until then IFD can make builds really slow when many projects use it at the same time.
At Nixcademy, we help teams use IFD the right way. We can study your existing projects, requirements, and infrastructure and help your teams use IFD without making your builds slow. With a proper set of guides everyone in your organization knows when to use IFD and when not to.
