FODONUTs: Fixed-Output Derivations for Operating Network-Utilizing Tests

2023-07-24

fodonut

Nix provides two main types of derivations:

  1. normal derivations
  2. fixed-output derivations (FODs)

The big difference is that fixed-output derivations allow you to access the internet from within the Nix build sandbox. However, they require you to give the hash of the derivation output in advance.

Aside from some exotic use-cases, fixed-output derivations are generally used to download tarballs and other files from the internet. Here's an example of a fixed-output derivation you may be familiar with:

fetchurl {
  url = "http://www.example.org/hello-1.0.tar.gz";
  sha256 = "0v6r3wwnsk5pdjr188nip3pjgn1jrn5pc5ajpcfy6had6b3v4dwm";
}

This derivation will download the tarball http://www.example.org/hello-1.0.tar.gz. Note that this tarball has to have the sha256 hash 0v6r3wwnsk5pdjr188nip3pjgn1jrn5pc5ajpcfy6had6b3v4dwm, or the derivation will fail after building.

Aside from downloading files from the internet, there is one other good use of fixed-output derivations: running a test (or a whole test-suite) that needs access to the internet. In this article I'll refer to these types of derivations as fixed-output derivations for operating network-utilizing tests (FODONUTs1).

Here's an example of a FODONUT. Imagine you're writing a derivation to build wget. After building wget, you naturally want to test that wget works correctly. The obvious way to do this is use wget to try to download a file from the internet. However, in a normal derivation, you can't access the internet. One way to work around this is to create a separate fixed-output derivation to run the wget tests2. If wget hasn't been built correctly, and it can't download files from the internet correctly, you can fail the Nix build.

If you're familiar with Nix, you'll immediately see two big problems here:

  1. FODs require the derivation output to be hashed. But in this example, what should we output? And what will its hash be?
  2. Normally, FODs don't get rebuilt unless their output hash changes.

This article discusses the solutions to these problems.

Nix Code in this Article

You can find all the Nix code for this article on GitHub. If you want to follow along, I recommend cloning this repo:

https://github.com/cdepillabout/example-fodonuts

This article has been written against nixpkgs-unstable from 2023-07-19, commit 57695599bdc4f7. But the Nix code here is relatively simple, and will likely work against any recent version of Nixpkgs.

A Review of Fixed-Output Derivations

First, let's have a short refresher on fixed-output derivations (FODs).

As stated above, FODs have two main differences to normal derivations:

  1. You're allowed to access the internet.
  2. You have to provide the hash of the file output from the derivation.

Let's take a look at an example of a trivial FOD:

fod-example-simple/default.nix:

{ stdenv }:

stdenv.mkDerivation {
  name = "fod-example-simple";

  # This is the sha256 hash for the string "hello world", which is output upon
  # this derivation building.
  outputHash = "sha256-qUiQTy8PR5uPgZdpSzAYSw0u0cHNKh7A+4XSmaGSpEc=";
  outputHashMode = "flat";
  outputHashAlgo = "sha256";

  buildCommand = ''
    echo 'hello world' > $out
  '';
}

The only thing this derivation does is output the string "hello world". We can try building this from the example-fodonuts repo:

$ nix build -L .#fod-example-simple
$ cat ./result
hello world

The main thing to notice here is that we must specify the hash of the the output string "hello world". If you try building this by specifying an incorrect hash, the build will fail.

Here's what the same derivation would look like with an incorrect hash:

{ stdenv }:

stdenv.mkDerivation {
  name = "fod-example-simple";

  # This is an _incorrect_ sha256 hash for the string "hello world".
  outputHash = "sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=";
  outputHashMode = "flat";
  outputHashAlgo = "sha256";

  buildCommand = ''
    echo 'hello world' > $out
  '';
}

Here's the error you get when you try to build:

$ nix build -L .#fod-example-simple
error: hash mismatch in fixed-output derivation '/nix/store/wadb5ayi877q4jzqi8abk4jiiycgdhws-fod-example-simple.drv':
         specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
            got:    sha256-qUiQTy8PR5uPgZdpSzAYSw0u0cHNKh7A+4XSmaGSpEc=

Let's take a look at a more substantial fixed-output derivation. This uses curl to download a tarball from the internet:

fod-example/default.nix:

{ cacert, curl, lib, stdenv }:

stdenv.mkDerivation {

  name = "fod-example";

  # This is the sha256 hash for the string "success", which is output upon this
  # test succeeding.
  outputHash = "sha256-7k4HUasQi22k9HxS2hh9UXfcNx8PUSp8quxUNOcRwJE=";
  outputHashMode = "flat";
  outputHashAlgo = "sha256";

  nativeBuildInputs = [ curl ];

  # Needed for people using Nix behind a proxy.
  impureEnvVars = lib.fetchers.proxyImpureEnvVars;

  buildCommand = ''
    curl=(
      curl --location --max-redirs 20 --retry 3 --disable-epsv
      --cookie-jar cookies --user-agent "curl " --insecure
    )

    # Make sure curl can access HTTPS sites, like GitHub.
    export SSL_CERT_FILE="${cacert}/etc/ssl/certs/ca-bundle.crt"

    "''${curl[@]}" \
      "https://github.com/BurntSushi/ripgrep/releases/download/13.0.0/ripgrep-13.0.0-x86_64-unknown-linux-musl.tar.gz" \
      > $out
  '';
}

Here's how to build it:

$ nix build -L .#fod-example
$ file -L ./result
./result: gzip compressed data, last modified: Sat Jun 12 12:32:02 2021, from Unix, original size modulo 2^32 5601280

You can see that the output file is actually a gzip, as expected.

There are a couple interesting things to notice here:

  1. If you re-run the nix build command, nothing happens. The file doesn't get re-downloaded. This is because Nix knows that it doesn't need to rebuild FODs if they have been already built once.

    How does Nix know if it has already built an FOD? It looks at a combination of the derivation name and hash values. In this case, the name is fod-example, and the hash is sha256-7k4HUasQi22k9HxS2hh9UXfcNx8PUSp8quxUNOcRwJE=.

    If you change either one of these values, the derivation will be rebuilt. For instance, try changing the name to foo-example-2, and re-running nix build -L. The file should be downloaded again, and a new output is created in the Nix store.

    If you change the hash value, the derivation will also be rebuilt. Although, like we saw above, if the hash is incorrect the build will fail.

  2. A consequence of the previous point is that if you keep the name and hash values the same, but change anything else in the derivation, it will not be rebuilt. For instance, try changing the URL in the buildCommand to https://github.com/BurntSushi/ripgrep/releases/download/12.0.0/ripgrep-12.0.0-x86_64-unknown-linux-musl.tar.gz, keep the name and hash values the same, and re-run nix build.

    You will see that nix build finishes cleanly, without rebuilding the derivation, or downloading the new version 12.0.0 tarball. Nix realizes it has already built a derivation with the same name and hash, and just returns that output. Most people new to Nix are surprised by this behavior!3

  3. There are a couple tricky things in this derivation, like setting impureEnvVars, SSL_CERT_FILE, and various curl options. These are all necessary for robustly using curl from within the Nix sandbox, but they are not particuarly interesting. This article won't go into detail about why they are needed.

Now that you have an understanding of FODs, let's see how we can use them to run tests that connect to the internet.

FODONUTs: Fixed-Output Derivations for Operating Network-Utilizing Tests

Imagine we have a program we're building with Nix. Let's use wget as a running example. After building wget, we want to test that wget can correctly download files from the internet. This section explains how we can leverage fixed-output derivations for operating network-utilizing tests (FODONUTs).

We'll work through a couple different variations on this theme, and work towards writing a an example FODONUT that is widely useful.

My wget

This section will use wget as an example.

Let's create a separate wget derivation with a custom name. This is a Nixpkgs overlay:

overlay.nix:

final: prev: {
  my-wget = final.wget.overrideAttrs { pname = "my-wget"; };
}

You can try building and using this from the example-fodonuts repo:

$ nix build -L .#my-wget
$ ./result/bin/wget http://google.com/ -O index.html
$ head index.html
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en"><head>
...

You can see that this my-wget derivation is able to be built correctly. By running it on the command line, we can see that it appears to be working correctly.

However, we'd still like to be able to run tests for it in a Nix derivation.

Attempt 1: a Normal Derivation

As a first attempt at writing a derivation for running tests for wget, lets see what happens if we just use a normal derivation (non-FOD):

my-wget-tests-attempt-1-normal-derivation/default.nix:

{ my-wget, lib, stdenv }:

stdenv.mkDerivation {

  name = "my-wget-tests-attempt-1-normal-derivation";

  nativeBuildInputs = [ my-wget ];

  # Needed for people using Nix behind a proxy.
  impureEnvVars = lib.fetchers.proxyImpureEnvVars;

  buildCommand = ''
    wget http://www.google.com -O index.html

    echo success > "$out"
  '';
}

This test just uses wget to download the homepage of http://www.google.com.

You can try building this derivation from the example-fodonuts repo:

$ nix build -L .#my-wget-tests-attempt-1-normal-derivation
my-wget-tests-attempt> --2023-07-20 08:27:16--  http://google.com/
my-wget-tests-attempt> Resolving google.com (google.com)... failed: Temporary failure in name resolution.
my-wget-tests-attempt> wget: unable to resolve host address 'google.com'
error: builder for '/nix/store/aidsrs8xpb32dr37pfqlsca2ag94mm1s-my-wget-tests-attempt-1-normal-derivation.drv' failed with exit code 4

You can see here that because of the Nix build sandbox, wget is not able to connect to the internet. In order to work around this, we need to use an FOD4.

Attempt 2: A Fixed-Output Derivation

As a second attempt, let's change to using an FOD:

my-wget-tests-attempt-2-fod/default.nix:

{ my-wget, lib, stdenv }:

stdenv.mkDerivation {

  name = "my-wget-tests-attempt-2-fod";

  # This is the sha256 hash for the string "success", which is output upon this
  # test succeeding.
  outputHash = "sha256-gbK9TqmMjbZlVPvI12N6GmmhMPMx/rcyt1yqtMSGj9U=";
  outputHashMode = "flat";
  outputHashAlgo = "sha256";

  nativeBuildInputs = [ my-wget ];

  # Needed for people using Nix behind a proxy.
  impureEnvVars = lib.fetchers.proxyImpureEnvVars;

  buildCommand = ''
    wget http://www.google.com -O index.html

    echo success > "$out"
  '';
}

Let's try building this:

$ nix build -L .#my-wget-tests-attempt-2-fod
my-wget-tests-attempt> --2023-07-20 09:33:41--  http://www.google.com/
my-wget-tests-attempt> Resolving www.google.com (www.google.com)... 172.217.161.36, 2404:6800:4004:826::2004
my-wget-tests-attempt> Connecting to www.google.com (www.google.com)|172.217.161.36|:80... connected.
my-wget-tests-attempt> HTTP request sent, awaiting response... 200 OK
my-wget-tests-attempt> Length: unspecified [text/html]
my-wget-tests-attempt> Saving to: 'index.html'
my-wget-tests-attempt> 2023-07-20 09:33:41 (708 KB/s) - 'index.html' saved [20892]

Success! Since we're building in an FOD, wget is now able to connect to the internet. After building this once, if we try rebuilding, nothing happens, since the result is cached in our Nix store.

However, we have a problem here. Even if we change the my-wget derivation, this my-wget-tests-attempt-2-fod derivation won't be rebuilt. Let's look at an example of this.

Try changing my-wget to be defined like the following:

final: prev: {
  my-wget =
    final.wget.overrideAttrs {
      pname = "my-wget";
      postInstall = ''
        rm $out/bin/wget
      '';
    };
}

We've introduced an obvious problem to the my-wget derivation. It builds correctly, but we've accidentally deleted the output wget binary we're expecting to use! This is something that should definitely be caught by our tests, so lets try running them one more time.

$ nix build -L .#my-wget-tests-attempt-2-fod

Hmm, this succeeds. We're not seeing a test failure like we expect. What's going on here?

As explained above, if you keep the name and hash values the same, but change anything else in the derivation (including things like our my-wget dependency), the derivation will not be rebuilt when running nix build.

When you run nix build, Nix sees that it has already built a derivation with the same name and hash, and just uses that result.

As an example, try changing the name of the derivation from

name = "my-wget-tests-attempt-2-fod";

to

name = "my-wget-tests-attempt-2-fod-foobar";

and rebuilding. You can see that it tries to rebuild the derivation, and fails, as expected:

$ nix build -L .#my-wget-tests-attempt-2-fod
my-wget-tests-attempt> /nix/store/fzb9wy1yz0hn69vxw12954szvrjnjjgk-stdenv-linux/setup: line 1559: wget: command not found
error: builder for '/nix/store/k79v1rh11g80sfdmhx7jsnijka081z5x-my-wget-tests-attempt-2-fod-foobar.drv' failed with exit code 127;

We would ideally like some way to have the my-wget-tests-attempt-2 derivation to be rebuilt whenever the my-wget input changes. Let's see how we can accomplish this.

Attempt 3: A Fixed-Output Derivation with a Special Name

There is an easy way to make sure the tests are actually run when the underlying my-wget derivation changes. Put the output hash of my-wget into the name of tests. Let's look at an example of this:

my-wget-tests-attempt-3-fod-special-name/default.nix:

{ my-wget, lib, stdenv }:

let
  my-wget-hash = builtins.hashString "md5" my-wget.outPath;
in

stdenv.mkDerivation {

  name = "my-wget-tests-attempt-3-fod-special-name-${my-wget-hash}";

  # This is the sha256 hash for the string "success", which is output upon this
  # test succeeding.
  outputHash = "sha256-gbK9TqmMjbZlVPvI12N6GmmhMPMx/rcyt1yqtMSGj9U=";
  outputHashMode = "flat";
  outputHashAlgo = "sha256";

  nativeBuildInputs = [ my-wget ];

  # Needed for people using Nix behind a proxy.
  impureEnvVars = lib.fetchers.proxyImpureEnvVars;

  buildCommand = ''
    wget http://www.google.com -O index.html

    echo success > "$out"
  '';
}

Here, my-wget.outPath will look something like:

$ nix repl
nix-repl> :lf ./.
nix-repl> packages.x86_64-linux.my-wget.outPath
"/nix/store/0npkkrijy52bhpcb1g7r2h9jvgv8ym17-my-wget-1.21.4"

This is the full path to the derivation output in the Nix store. Hashing this gives us a value like the following:

nix-repl> builtins.hashString "md5" packages.x86_64-linux.my-wget.outPath
"6311bc8ae58b49e918900025ead0d10b"

So the full name of the test becomes:

nix-repl> packages.x86_64-linux.my-wget-tests-attempt-3-fod-special-name.name
"my-wget-tests-attempt-3-fod-special-name-6311bc8ae58b49e918900025ead0d10b"

Let's build this test and confirm it succeeds:

$ nix build -L .#my-wget-tests-attempt-3-fod-special-name
my-wget-tests-attempt> --2023-07-22 07:11:29--  http://www.google.com/
my-wget-tests-attempt> Resolving www.google.com (www.google.com)... 142.251.42.132, 2404:6800:4004:825::2004
my-wget-tests-attempt> Connecting to www.google.com (www.google.com)|142.251.42.132|:80... connected.
my-wget-tests-attempt> HTTP request sent, awaiting response... 200 OK
my-wget-tests-attempt> Length: unspecified [text/html]
my-wget-tests-attempt> Saving to: 'index.html'
my-wget-tests-attempt> index.html              [ <=>                ]  18.90K  --.-KB/s    in 0.02s
my-wget-tests-attempt> 2023-07-22 07:11:29 (762 KB/s) - 'index.html' saved [19355]

Just to make sure, let's change the my-wget derivation again so that it is broken:

final: prev: {
  my-wget =
    final.wget.overrideAttrs {
      pname = "my-wget";
      postInstall = ''
        rm $out/bin/wget
      '';
    };
}

Let's confirm in the repl that the hash my-wget, used in the name of the test, has changed:

$ nix repl
nix-repl> :lf ./.
nix-repl> packages.x86_64-linux.my-wget-tests-attempt-3-fod-special-name.name
"my-wget-tests-attempt-3-fod-special-name-0efdb406714d5f4f279d23c888870f1b"

Great, it has changed! Now when we try to rebuild the test, it should actually be rebuilt:

$ nix build -L .#my-wget-tests-attempt-3-fod-special-name
my-wget-tests-attempt> /nix/store/fzb9wy1yz0hn69vxw12954szvrjnjjgk-stdenv-linux/setup: line 1559: wget: command not found
error: builder for '/nix/store/4vsk6n3lyq6q5l9rrv3d8zyq6piklq4g-my-wget-tests-attempt-3-fod-special-name-0efdb406714d5f4f279d23c888870f1b.drv' failed with exit code 127;

Success! Err, failure... but that's what we were hoping for!

So this seems like it is working well, and doing what we expect. This example is the heart of a fixed-output derivation for operating network-utilizing tests (FODNUT):

What is a FODONUT?

A FODONUT is a fixed-output derivation with a special name. Take a hash of the derivation containing the binary you want to test, and put it in the name of the FOD. You'll be able to access the network in the tests, and the derivation will be correctly rebuilt whenever the main dependency changes.

There is, however, still one problem here5.

Imagine we want to change the test derivation itself. Right now it is downloading an HTML file from http://www.google.com, but we want to change it to instead download an HTML file from http://www.example.com. Let's make that change:

{ my-wget, lib, stdenv }:

let
  my-wget-hash = builtins.hashString "md5" my-wget.outPath;
in

stdenv.mkDerivation {

  name = "my-wget-tests-attempt-3-fod-special-name-${my-wget-hash}";

  # This is the sha256 hash for the string "success", which is output upon this
  # test succeeding.
  outputHash = "sha256-gbK9TqmMjbZlVPvI12N6GmmhMPMx/rcyt1yqtMSGj9U=";
  outputHashMode = "flat";
  outputHashAlgo = "sha256";

  nativeBuildInputs = [ my-wget ];

  # Needed for people using Nix behind a proxy.
  impureEnvVars = lib.fetchers.proxyImpureEnvVars;

  buildCommand = ''
    wget http://www.example.com -O index.html

    echo success > "$out"
  '';
}

And try rebuilding the test:

$ nix build -L .#my-wget-tests-attempt-3-fod-special-name

Hmm, it doesn't rebuild. This is because we changed the test derivation itself, but we haven't changed the my-wget derivation at all. Again, Nix sees that it has already built a FOD with this name and hash, so doesn't try to rebuild.

The solution to this problem leads us to a full FODONUT.

Attempt 4: A Full FODONUT

A straightforward solution to this is to just hash the full test script, and stick it into the name of the test derivation. Here's an example of what this might look like:

{ my-wget, lib, stdenv }:

let
  buildCommand = ''
    wget http://www.google.com -O index.html

    echo success > "$out"
  '';

  value-to-hash = buildCommand + my-wget.outPath;

  test-name-hash = builtins.hashString "md5" value-to-hash;
in

stdenv.mkDerivation {

  name = "my-wget-tests-attempt-4-fodonut-${test-name-hash}";

  # This is the sha256 hash for the string "success", which is output upon this
  # test succeeding.
  outputHash = "sha256-gbK9TqmMjbZlVPvI12N6GmmhMPMx/rcyt1yqtMSGj9U=";
  outputHashMode = "flat";
  outputHashAlgo = "sha256";

  nativeBuildInputs = [ my-wget ];

  # Needed for people using Nix behind a proxy.
  impureEnvVars = lib.fetchers.proxyImpureEnvVars;

  inherit buildCommand;
}

Here, you can see that we are hashing both the full buildCommand, as well as the output path of my-wget. Let's confirm this builds:

$ nix build -L .#my-wget-tests-attempt-4-fodonut
my-wget-tests-attempt> --2023-07-22 08:20:23--  http://www.google.com/
my-wget-tests-attempt> Resolving www.google.com (www.google.com)... 216.58.220.100, 2404:6800:4004:80a::2004
my-wget-tests-attempt> Connecting to www.google.com (www.google.com)|216.58.220.100|:80... connected.
my-wget-tests-attempt> HTTP request sent, awaiting response... 200 OK
my-wget-tests-attempt> Length: unspecified [text/html]
my-wget-tests-attempt> Saving to: 'index.html'
my-wget-tests-attempt> index.html              [ <=>                ]  18.88K  --.-KB/s    in 0.03s
my-wget-tests-attempt> 2023-07-22 08:20:23 (730 KB/s) - 'index.html' saved [19334]

Go back into the derivation and try changing the URL from http://www.google.com to http://www.example.com, and rebuilding:

$ nix build -L .#my-wget-tests-attempt-4-fodonut
my-wget-tests-attempt> --2023-07-22 08:24:36--  http://www.example.com/
my-wget-tests-attempt> Resolving www.example.com (www.example.com)... 93.184.216.34, 2606:2800:220:1:248:1893:25c8:1946
my-wget-tests-attempt> Connecting to www.example.com (www.example.com)|93.184.216.34|:80... connected.
my-wget-tests-attempt> HTTP request sent, awaiting response... 200 OK
my-wget-tests-attempt> Length: 1256 (1.2K) [text/html]
my-wget-tests-attempt> Saving to: 'index.html'
my-wget-tests-attempt> index.html          100%[===================>]   1.23K  --.-KB/s    in 0s
my-wget-tests-attempt> 2023-07-22 08:24:37 (179 MB/s) - 'index.html' saved [1256/1256]

It is rebuilt! Pefect!

Key Takeaways for FODONUTs

So what really are fixed-output derivation for operating network-utilizing tests (FODONUTs)?

FODONUTs are the following:

  1. Fixed-output derivations that output a simple, known string, like "success".

  2. The derivation runs a test that requires access to the internet (so it won't build correctly in a normal sandboxed derivation).

  3. FODONUTs have a hash in their name. The hash is carefully crafted to make sure rebuilds happen as a user might expect.

FODONUTs are quite useful for tests, but have a few downsides as well:

  1. They can really only output one bit of information: whether or not the tests succeed. It is not possible for the tests to output some sort of "test report", since FODs aren't able to output anything other than a previously known hashed value6.

  2. Once a test succeeds, it won't run again until its name or hash changes. While this is exactly what we want in many scenarios, there are certain kinds of tests where the exact state of the data we are accessing on the internet has a big affect on the outcome of the test. Data from the internet may change frequently, and cause frequent test failures.

    For instance, with the my-wget example, we are making sure wget can download the front page of Google. In this case, Google is quite reliable. If wget is not able to download the front page of Google, is is likely a problem with wget, not Google.7

    As a rule of thumb, FODONUTs only make sense for tests that download data that isn't going to frequently change. Or, at least, data that won't change in a way that causes the tests to fail. If you're using a FODONUT, and your tests frequently fail because of changes to data from the internet, you should rethink whether FODONUTs are right for your use-case8.

  3. Sometimes you have to be quite careful exactly what information ends up in the the name of a FODONUT. You want to make sure this causes the derivation to be rebuilt only when it should be rebuilt. You don't want too many rebuilds, and you don't want your tests to accidentally not get run when they should.

Examples of FODONUTs in the Wild

I know of a few examples of FODONUTs in the wild:

  1. Stack / Hpack version test9 in Nixpkgs

    This is a test to make sure that the version of the hpack dependency used by stack in Nixpkgs matches the same version as upstream. This test downloads a statically-linked stack binary from GitHub, and compares versions of the hpack dependency with the stack from Nixpkgs.

    There are two interesting things to notice here:

    1. The name of the test derivation contains the stack and hpack version numbers, so if the stack or hpack version is updated in Nixpkgs, this test derivation will be rebuilt.

      The Haskell infrastructure in Nixpkgs is setup so that stack and hpack may be updated semi-automatically, so this test derivation will helpfully catch when stack or hpack are updated to versions that are incompatible with upstream releases.

    2. You can see that the test script uses the cacert derivation. However, changes in the cacert derivation are not likely to cause this test to start failing, so the cacert derivation is explicitly not included anywhere in the hash added to the test name. We have to be somewhat careful with how we write the test script.

    You may be wondering why this couldn't just be split into a FOD that downloads the upstream statically-linked stack binary, and a separate, normal derivation that performs the test.

    It could be split-up like that, but then we'd have to manually bump the hash of the upstream stack binary. Writing this as a FODONUT saves us the trouble of having to manually update the hash.

  2. The tests for the fetchers in Nixpkgs, like fetchurl.

    While these aren't exactly FODONUTs, they do share some of the same tactics.

  3. evil-nix

    This doesn't technically use FODONUTs, but it does contain a FOD with a carefully constructed name.

  4. The repro-get pkg-version test in Nixpkgs

    This was authored by Matthew Croughan and Robert Hensing.

I imagine there are more examples of FODONUTs being used in other projects and proprietary repos. If you have an example of a FODONUT, please share!

Conclusion

While I'm sure I'm not the first to come up with the idea of using FODs to run tests that access the internet, I haven't seen any other blog posts talking about the idea. This post is an attempt to write up an easy-to-understand explanation of how FODONUTS work.

FODONUTs are quite a helpful technique for running tests within Nix derivations. When building software with Nix, it is very common to want to get data from the internet to use in tests, but it is often too difficult or annoying to download that data with a fixed-output derivation. FODONUTs can be a nice compromise in this situation.

Footnotes


  1. I read this as "foe donuts", or "faux donuts".↩︎

  2. Another common solution is to standup a webserver like Nginx on localhost within a normal Nix derivation, and make requests to it with wget. The advantage with this is that everything happens within a normal Nix derivation, and in theory it is more reproducible than accessing something out on the internet. The disadvantage is that you must download all resources required for the tests using FODs. While this is possible for some types of tests, for others it is quite a big limitation.↩︎

  3. In theory, if you've already output one thing with a given name/hash, it doesn't matter if you change the build steps, you'll still get the same value if you rebuild it.

    In practice, I'm not sure how good of a decision this is, since it can be quite surprising to users when they update the build steps in an FOD, but forget to change the hash. But it is also nice in some ways.↩︎

  4. You can also work around this by disabling the build sandbox:

    $ nix build -L .#my-wget-tests-attempt-1-normal-derivation --option sandbox false

    But this requires you to run the build as a trusted-user, which is quite limiting, and not something that is allowed in Nixpkgs.↩︎

  5. There are a couple other things that we could talk about as well, which aren't important enough for the main post:

    1. Is it safe to use an MD5 hash here?

      Yes. There are no security implications here, so MD5 is alright. It is extremely unlikely we get a collision, and it is really not the end of the world if there just happens to be a collision.

    2. Is there a better method than hashing the full path of the my-wget output?

      Yes. We could just take the hash from the path of the my-wget output, and use that. For example, the full path of the my-wget output is:

      $ nix path-info .#my-wget
      /nix/store/qn523zxch23jsa5jpc4b2fhr25zh85kn-my-wget-1.21.4

      Instead of MD5 hashing this path, we could just directly use the hash qn523zxch23jsa5jpc4b2fhr25zh85kn. The big advantage of this is that we don't have to compute an MD5 hash again, and for people that use an alternative /nix/store path, they will get the same value.

      There really isn't any downside here, except it is slightly more difficult to code up and explain.

    ↩︎
  6. Keep in mind that Nix builds have a concept of a "build log". It is possible to output test failures to the build log, and inspect them at a later time.

    For instance, try changing the my-wget-tests-attempt-4-fodonut derivation to download from a domain lalalathisdomaindoesnotexist.com instead of google.com. Here's what happens when you run this test:

    $ nix build -L .#my-wget-tests-attempt-4-fodonut
    warning: Git tree '/home/illabout/git/example-fodonuts' is dirty
    my-wget-tests-attempt> --2023-07-22 11:30:59--  http://www.lalalathisdomaindoesnotexist.com/
    my-wget-tests-attempt> Resolving www.lalalathisdomaindoesnotexist.com (www.lalalathisdomaindoesnotexist.com)... failed: Name or service not known.
    my-wget-tests-attempt> wget: unable to resolve host address 'www.lalalathisdomaindoesnotexist.com'
    error: builder for '/nix/store/inw2snkn1cl7v6x83x0z03006q4rn8q7-my-wget-tests-attempt-4-fodonut-6c62f59d6308b771c9a6353f23f518e0.drv' failed with exit code 4;

    We can of course see the build failure here, and Nix also has a built-in way of viewing the build logs for this build failure:

    $ nix log /nix/store/inw2snkn1cl7v6x83x0z03006q4rn8q7-my-wget-tests-attempt-4-fodonut-6c62f59d6308b771c9a6353f23f518e0.drv
    --2023-07-22 11:30:59--  http://www.lalalathisdomaindoesnotexist.com/
    Resolving www.lalalathisdomaindoesnotexist.com (www.lalalathisdomaindoesnotexist.com)... failed: Name or service not known.
    wget: unable to resolve host address 'www.lalalathisdomaindoesnotexist.com'
    ↩︎
  7. In practice, this likely depends on things like the stability of your internet, what sort of data you are downloading and using in your tests, etc.↩︎

  8. Another way to think about this: if your FODONUTs frequently fail because of changes to data from the internet, you likely want tests that are able to run more frequently than just whenever the underlying derivation changes.

    In our my-wget example, in practice, we might be really worried about the compatibility between my-wget and the upstream website we're trying to download from. However, the my-wget derivation might not change much, so the tests won't actually be run that often. In this case, a FODONUT might not be the best choice, and you should really try to find some way of running tests outside of a Nix derivation.

    If, for some reason, you don't have any way to run tests except as Nix derivations, you might be able to force rebuilds of your FODONUT by using things like builtins.currentTime in the name. Or possibly just turning off Nix's build-time sandox.↩︎

  9. I originally wrote this test.↩︎

tags: nixos