Ekpkgs: Streamlining Nixpkgs and Flakes for a Smoother Experience

Ekpkgs aims to address the “awkwardness budget” as decribed in the first post about ekpkgs by introducing a more consistent and intuitive approach to working with nixpkgs and flakes. This post explores how ekpkgs plans to simplify overlays, enable extensible package scopes, and streamline development shells.

Simplifying Overlay usage

In my opinion, overlays are the best distinguishing characteristic of nixpkgs. Overlays empower users to extend and customize the package set to fit their needs. They let you treat your software as if it were part of nixpkgs itself. However, the current overlay system has limitations. For example, overlays typically target the top-level pkgs scope, which can lead to complex and error-prone code when modifying specific ecosystems like Python.

Consider the official recommendation for overriding a python package using an overlay:

# According to official docs, this is preferred for a python interpreter:
self: super: {
  python = super.python.override {
    packageOverrides = python-self: python-super: {
      twisted = python-super.twisted.overrideAttrs (oldAttrs: {
        src = super.fetchPypi {
          pname = "Twisted";
          version = "19.10.0";
          hash = "sha256-c5S6fycq5yKnTz2Wnc9Zm8TvCTvDkgOHSKSQ8XJKUV0=";
          extension = "tar.bz2";
        };
      });
    };

    # The next line was actually missing from the example
    # but it ensures that if you reference python.pkgs from within
    # the scope you get same package scope.
    # Otherwise it will get you the unaltered package scope:
    self = self.python;
  };
}

Unique to python, you also have a special attrbute at the pkgs scope for just declaring python overlays which is respected by all python interpreters:

final: prev: {
  pythonPackagesExtensions = prev.pythonPackagesExtensions ++ [
    (
      python-final: python-prev: {
        foo = python-prev.foo.overridePythonAttrs (oldAttrs: {
          # ...
        });
      }
    )
  ];
}

This setup is less than ideal. The Python-specific syntax doesn’t translate to other ecosystems, and forgetting details like self = self.python can lead to inconsistent behavior. With multiple ways to apply overlays, debugging unexpected packaging issues becomes a scavenger hunt.

Ekpkgs proposes a more unified solution by exposing overlays as pkgs module options when importing the package set. For example:

pkgs = import ekapkgs {
  config.overlays = {
    pkgs = import ./overlays/top-level.nix;
    python = import ./overlays/python.nix;
    haskell = import ./overlays/haskell.nix;
  };
}

In this model, overlays are applied at their respective scopes (e.g., Python or Haskell), ensuring consistency across all variants of that scope. No more targeting specific Python versions manually. By leveraging the Nix module system, ekpkgs also enables tools like mkBefore, mkAfter, and mkForce, giving users fine-grained control over overlay precedence when combining multiple overlays.

Empowering Extensible Package Scopes

This overlay overhaul paves the way for extensible package scopes, allowing users and maintainers to seamlessly integrate new ecosystems. Ekpkgs will use this internally to compose packages from multiple repositories, but the approach is general enough for anyone to adopt. The pkgsModules proposal further enhances this approach by allowing modules from many repositories to be combined in a uniform manner:

let
  pkgsModule = {
    overlays = {
      pkgs = import ./overlays/top-level.nix;
      python = import ./overlays/python.nix;
      haskell = import ./overlays/haskell.nix;
    };
  };

  pkgs = import ekapkgs {
    modules = [ pkgsModule ];
  };
in
  ...

Although this seems like a minor refactor; the combination of allowing other modules in evaluation of pkgs.config and easy extension of package sets allows for a very ergonomic way to combine many overlay ecosystems. Similarly, if an overlay has config options which should be known before attempting a package (e.g. cudaCompat) then this also allows for corepkgs to be unaware of these options but can be added later.

To demonstrate this usage more broadly, this will likely be the flake for ekapkgs:

{
  inputs = {
    corepkgs.url = ...;
    python.url = ...;
    node.url = ...;
    haskell.url = ...;
    cuda.url = ...;
    vim.url = ...;
    emacs.url = ...;
    ...
  };

  outputs = { corepkgs, ... }@inputs:
    # Just a teaser for now, ;)
    corepkgs.mkFlakeOutputs {
      # all pkgsModules from each input are listed here
      modules = [ ... ];
  };
}

This produces a .#legacyPackages.${system} with all overlays applied for each known system in ekapkgs.

Allow for dev shells to be generated from derivations

NOTE: This proposal is much less refined in scope and thought, and likely to change before being ratified, if ratified at all. In particular, it requires a fundamental change to how package scopes + stdenv works.

One common painpoint with devshells is, “I want a dev shell which is like my package but just slightly different”. For most, they will use something like mkShell; however, this has a few leaky abstractions. Namely, mkShell will bring in stdenv.cc as the wrapped compiler, so if your original package is using something like clangStdenv or buildRustPackage, you will suddenly have GCC instead of clang, environment variables are likely different, and many other details get lost in translation. This is similarly true for most other “mkDerivation helpers” such as buildGoModule, buildPythonPackage, or buildVimPlugin. Instead, we generally want to take something which works for a package release, then alter it slightly to be more approriate for development. In this effort, I would eventually like to add a toDevShell utility for all mkDerivation packages. This would work similarly to mkShell but it’s available to all derivations which use mkDerivation and all inputs are appended to the existing derivation.

# flake.nix usage
outputs = { corepkgs, self }@inputs:
  corepkgs.mkFlakeOutputs {
    modules = [ ./nix/pkgs-module.nix ];
    packages = pkgs: {
      default = pkgs.myPackage;
    };
    devShells = pkgs: {
      default = pkgs.myPackage.toDevShell( { clippy, rustfmt }: {
        # These are appended, rather than replacing the previous values
        nativeBuildInputs = [ clippy rustfmt ];
        # To set environment variables just for dev shell
        env.DEBUG = true;
      });
    };
  };

In the future, this model will likely also be extended to allow for setting build types to debug which are generally much faster to iterate with, provide debug symbols, and avoids scenarios where a release and debug built occurs (e.g. buildRustPackage + doCheck = true;).

Conclusion

Ekpkgs is a bold step toward reducing the friction of working with nixpkgs and flakes. By simplifying overlays, enabling extensible package scopes, and improving dev shell workflows, it aims to make Nix more approachable and consistent. While some proposals are still in draft, I desire a Nix ecosystem where power and flexibility doesn’t come at the cost of complexity.