2023-07-24
Nix provides two main types of derivations:
- normal derivations
- fixed-output derivations (FODs)
The big difference is that fixed-output derivations allow you to access the internet from within the Nix build sandbox. However, they require you to give the hash of the derivation output in advance.
Aside from some exotic use-cases, fixed-output derivations are generally used to download tarballs and other files from the internet. Here's an example of a fixed-output derivation you may be familiar with:
fetchurl {
url = "http://www.example.org/hello-1.0.tar.gz";
sha256 = "0v6r3wwnsk5pdjr188nip3pjgn1jrn5pc5ajpcfy6had6b3v4dwm";
}
This derivation will download the tarball http://www.example.org/hello-1.0.tar.gz
. Note that this tarball has to have the sha256 hash 0v6r3wwnsk5pdjr188nip3pjgn1jrn5pc5ajpcfy6had6b3v4dwm
, or the derivation will fail after building.
Aside from downloading files from the internet, there is one other good use of fixed-output derivations: running a test (or a whole test-suite) that needs access to the internet. In this article I'll refer to these types of derivations as fixed-output derivations for operating network-utilizing tests (FODONUTs1).
Here's an example of a FODONUT. Imagine you're writing a derivation to build wget
. After building wget
, you naturally want to test that wget
works correctly. The obvious way to do this is use wget
to try to download a file from the internet. However, in a normal derivation, you can't access the internet. One way to work around this is to create a separate fixed-output derivation to run the wget
tests2. If wget
hasn't been built correctly, and it can't download files from the internet correctly, you can fail the Nix build.
If you're familiar with Nix, you'll immediately see two big problems here:
- FODs require the derivation output to be hashed. But in this example, what should we output? And what will its hash be?
- Normally, FODs don't get rebuilt unless their output hash changes.
This article discusses the solutions to these problems.
Nix Code in this Article
You can find all the Nix code for this article on GitHub. If you want to follow along, I recommend cloning this repo:
https://github.com/cdepillabout/example-fodonuts
This article has been written against nixpkgs-unstable
from 2023-07-19, commit 57695599bdc4f7. But the Nix code here is relatively simple, and will likely work against any recent version of Nixpkgs.
A Review of Fixed-Output Derivations
First, let's have a short refresher on fixed-output derivations (FODs).
As stated above, FODs have two main differences to normal derivations:
- You're allowed to access the internet.
- You have to provide the hash of the file output from the derivation.
Let's take a look at an example of a trivial FOD:
fod-example-simple/default.nix
:
{ stdenv }:
stdenv.mkDerivation {
name = "fod-example-simple";
# This is the sha256 hash for the string "hello world", which is output upon
# this derivation building.
outputHash = "sha256-qUiQTy8PR5uPgZdpSzAYSw0u0cHNKh7A+4XSmaGSpEc=";
outputHashMode = "flat";
outputHashAlgo = "sha256";
buildCommand = ''
echo 'hello world' > $out
'';
}
The only thing this derivation does is output the string "hello world"
. We can try building this from the example-fodonuts repo:
$ nix build -L .#fod-example-simple
$ cat ./result
hello world
The main thing to notice here is that we must specify the hash of the the output string "hello world"
. If you try building this by specifying an incorrect hash, the build will fail.
Here's what the same derivation would look like with an incorrect hash:
{ stdenv }:
stdenv.mkDerivation {
name = "fod-example-simple";
# This is an _incorrect_ sha256 hash for the string "hello world".
outputHash = "sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=";
outputHashMode = "flat";
outputHashAlgo = "sha256";
buildCommand = ''
echo 'hello world' > $out
'';
}
Here's the error you get when you try to build:
$ nix build -L .#fod-example-simple
error: hash mismatch in fixed-output derivation '/nix/store/wadb5ayi877q4jzqi8abk4jiiycgdhws-fod-example-simple.drv':
specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
got: sha256-qUiQTy8PR5uPgZdpSzAYSw0u0cHNKh7A+4XSmaGSpEc=
Let's take a look at a more substantial fixed-output derivation. This uses curl
to download a tarball from the internet:
{ cacert, curl, lib, stdenv }:
stdenv.mkDerivation {
name = "fod-example";
# This is the sha256 hash for the string "success", which is output upon this
# test succeeding.
outputHash = "sha256-7k4HUasQi22k9HxS2hh9UXfcNx8PUSp8quxUNOcRwJE=";
outputHashMode = "flat";
outputHashAlgo = "sha256";
nativeBuildInputs = [ curl ];
# Needed for people using Nix behind a proxy.
impureEnvVars = lib.fetchers.proxyImpureEnvVars;
buildCommand = ''
curl=(
curl --location --max-redirs 20 --retry 3 --disable-epsv
--cookie-jar cookies --user-agent "curl " --insecure
)
# Make sure curl can access HTTPS sites, like GitHub.
export SSL_CERT_FILE="${cacert}/etc/ssl/certs/ca-bundle.crt"
"''${curl[@]}" \
"https://github.com/BurntSushi/ripgrep/releases/download/13.0.0/ripgrep-13.0.0-x86_64-unknown-linux-musl.tar.gz" \
> $out
'';
}
Here's how to build it:
$ nix build -L .#fod-example
$ file -L ./result
./result: gzip compressed data, last modified: Sat Jun 12 12:32:02 2021, from Unix, original size modulo 2^32 5601280
You can see that the output file is actually a gzip, as expected.
There are a couple interesting things to notice here:
If you re-run the
nix build
command, nothing happens. The file doesn't get re-downloaded. This is because Nix knows that it doesn't need to rebuild FODs if they have been already built once.How does Nix know if it has already built an FOD? It looks at a combination of the derivation
name
andhash
values. In this case, thename
isfod-example
, and thehash
issha256-7k4HUasQi22k9HxS2hh9UXfcNx8PUSp8quxUNOcRwJE=
.If you change either one of these values, the derivation will be rebuilt. For instance, try changing the name to
foo-example-2
, and re-runningnix build -L
. The file should be downloaded again, and a new output is created in the Nix store.If you change the
hash
value, the derivation will also be rebuilt. Although, like we saw above, if the hash is incorrect the build will fail.A consequence of the previous point is that if you keep the
name
andhash
values the same, but change anything else in the derivation, it will not be rebuilt. For instance, try changing the URL in thebuildCommand
tohttps://github.com/BurntSushi/ripgrep/releases/download/12.0.0/ripgrep-12.0.0-x86_64-unknown-linux-musl.tar.gz
, keep thename
andhash
values the same, and re-runnix build
.You will see that
nix build
finishes cleanly, without rebuilding the derivation, or downloading the new version 12.0.0 tarball. Nix realizes it has already built a derivation with the samename
andhash
, and just returns that output. Most people new to Nix are surprised by this behavior!3There are a couple tricky things in this derivation, like setting
impureEnvVars
,SSL_CERT_FILE
, and variouscurl
options. These are all necessary for robustly usingcurl
from within the Nix sandbox, but they are not particuarly interesting. This article won't go into detail about why they are needed.
Now that you have an understanding of FODs, let's see how we can use them to run tests that connect to the internet.
FODONUTs: Fixed-Output Derivations for Operating Network-Utilizing Tests
Imagine we have a program we're building with Nix. Let's use wget
as a running example. After building wget
, we want to test that wget
can correctly download files from the internet. This section explains how we can leverage fixed-output derivations for operating network-utilizing tests (FODONUTs).
We'll work through a couple different variations on this theme, and work towards writing a an example FODONUT that is widely useful.
My wget
This section will use wget
as an example.
Let's create a separate wget
derivation with a custom name. This is a Nixpkgs overlay:
You can try building and using this from the example-fodonuts repo:
$ nix build -L .#my-wget
$ ./result/bin/wget http://google.com/ -O index.html
$ head index.html
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en"><head>
...
You can see that this my-wget
derivation is able to be built correctly. By running it on the command line, we can see that it appears to be working correctly.
However, we'd still like to be able to run tests for it in a Nix derivation.
Attempt 1: a Normal Derivation
As a first attempt at writing a derivation for running tests for wget
, lets see what happens if we just use a normal derivation (non-FOD):
my-wget-tests-attempt-1-normal-derivation/default.nix
:
{ my-wget, lib, stdenv }:
stdenv.mkDerivation {
name = "my-wget-tests-attempt-1-normal-derivation";
nativeBuildInputs = [ my-wget ];
# Needed for people using Nix behind a proxy.
impureEnvVars = lib.fetchers.proxyImpureEnvVars;
buildCommand = ''
wget http://www.google.com -O index.html
echo success > "$out"
'';
}
This test just uses wget
to download the homepage of http://www.google.com.
You can try building this derivation from the example-fodonuts
repo:
$ nix build -L .#my-wget-tests-attempt-1-normal-derivation
my-wget-tests-attempt> --2023-07-20 08:27:16-- http://google.com/
my-wget-tests-attempt> Resolving google.com (google.com)... failed: Temporary failure in name resolution.
my-wget-tests-attempt> wget: unable to resolve host address 'google.com'
error: builder for '/nix/store/aidsrs8xpb32dr37pfqlsca2ag94mm1s-my-wget-tests-attempt-1-normal-derivation.drv' failed with exit code 4
You can see here that because of the Nix build sandbox, wget
is not able to connect to the internet. In order to work around this, we need to use an FOD4.
Attempt 2: A Fixed-Output Derivation
As a second attempt, let's change to using an FOD:
my-wget-tests-attempt-2-fod/default.nix
:
{ my-wget, lib, stdenv }:
stdenv.mkDerivation {
name = "my-wget-tests-attempt-2-fod";
# This is the sha256 hash for the string "success", which is output upon this
# test succeeding.
outputHash = "sha256-gbK9TqmMjbZlVPvI12N6GmmhMPMx/rcyt1yqtMSGj9U=";
outputHashMode = "flat";
outputHashAlgo = "sha256";
nativeBuildInputs = [ my-wget ];
# Needed for people using Nix behind a proxy.
impureEnvVars = lib.fetchers.proxyImpureEnvVars;
buildCommand = ''
wget http://www.google.com -O index.html
echo success > "$out"
'';
}
Let's try building this:
$ nix build -L .#my-wget-tests-attempt-2-fod
my-wget-tests-attempt> --2023-07-20 09:33:41-- http://www.google.com/
my-wget-tests-attempt> Resolving www.google.com (www.google.com)... 172.217.161.36, 2404:6800:4004:826::2004
my-wget-tests-attempt> Connecting to www.google.com (www.google.com)|172.217.161.36|:80... connected.
my-wget-tests-attempt> HTTP request sent, awaiting response... 200 OK
my-wget-tests-attempt> Length: unspecified [text/html]
my-wget-tests-attempt> Saving to: 'index.html'
my-wget-tests-attempt> 2023-07-20 09:33:41 (708 KB/s) - 'index.html' saved [20892]
Success! Since we're building in an FOD, wget
is now able to connect to the internet. After building this once, if we try rebuilding, nothing happens, since the result is cached in our Nix store.
However, we have a problem here. Even if we change the my-wget
derivation, this my-wget-tests-attempt-2-fod
derivation won't be rebuilt. Let's look at an example of this.
Try changing my-wget
to be defined like the following:
final: prev: {
my-wget =
final.wget.overrideAttrs {
pname = "my-wget";
postInstall = ''
rm $out/bin/wget
'';
};
}
We've introduced an obvious problem to the my-wget
derivation. It builds correctly, but we've accidentally deleted the output wget
binary we're expecting to use! This is something that should definitely be caught by our tests, so lets try running them one more time.
$ nix build -L .#my-wget-tests-attempt-2-fod
Hmm, this succeeds. We're not seeing a test failure like we expect. What's going on here?
As explained above, if you keep the name
and hash
values the same, but change anything else in the derivation (including things like our my-wget
dependency), the derivation will not be rebuilt when running nix build
.
When you run nix build
, Nix sees that it has already built a derivation with the same name
and hash
, and just uses that result.
As an example, try changing the name
of the derivation from
to
and rebuilding. You can see that it tries to rebuild the derivation, and fails, as expected:
$ nix build -L .#my-wget-tests-attempt-2-fod
my-wget-tests-attempt> /nix/store/fzb9wy1yz0hn69vxw12954szvrjnjjgk-stdenv-linux/setup: line 1559: wget: command not found
error: builder for '/nix/store/k79v1rh11g80sfdmhx7jsnijka081z5x-my-wget-tests-attempt-2-fod-foobar.drv' failed with exit code 127;
We would ideally like some way to have the my-wget-tests-attempt-2
derivation to be rebuilt whenever the my-wget
input changes. Let's see how we can accomplish this.
Attempt 3: A Fixed-Output Derivation with a Special Name
There is an easy way to make sure the tests are actually run when the underlying my-wget
derivation changes. Put the output hash of my-wget
into the name of tests. Let's look at an example of this:
my-wget-tests-attempt-3-fod-special-name/default.nix
:
{ my-wget, lib, stdenv }:
let
my-wget-hash = builtins.hashString "md5" my-wget.outPath;
in
stdenv.mkDerivation {
name = "my-wget-tests-attempt-3-fod-special-name-${my-wget-hash}";
# This is the sha256 hash for the string "success", which is output upon this
# test succeeding.
outputHash = "sha256-gbK9TqmMjbZlVPvI12N6GmmhMPMx/rcyt1yqtMSGj9U=";
outputHashMode = "flat";
outputHashAlgo = "sha256";
nativeBuildInputs = [ my-wget ];
# Needed for people using Nix behind a proxy.
impureEnvVars = lib.fetchers.proxyImpureEnvVars;
buildCommand = ''
wget http://www.google.com -O index.html
echo success > "$out"
'';
}
Here, my-wget.outPath
will look something like:
$ nix repl
nix-repl> :lf ./.
nix-repl> packages.x86_64-linux.my-wget.outPath
"/nix/store/0npkkrijy52bhpcb1g7r2h9jvgv8ym17-my-wget-1.21.4"
This is the full path to the derivation output in the Nix store. Hashing this gives us a value like the following:
nix-repl> builtins.hashString "md5" packages.x86_64-linux.my-wget.outPath
"6311bc8ae58b49e918900025ead0d10b"
So the full name
of the test becomes:
nix-repl> packages.x86_64-linux.my-wget-tests-attempt-3-fod-special-name.name
"my-wget-tests-attempt-3-fod-special-name-6311bc8ae58b49e918900025ead0d10b"
Let's build this test and confirm it succeeds:
$ nix build -L .#my-wget-tests-attempt-3-fod-special-name
my-wget-tests-attempt> --2023-07-22 07:11:29-- http://www.google.com/
my-wget-tests-attempt> Resolving www.google.com (www.google.com)... 142.251.42.132, 2404:6800:4004:825::2004
my-wget-tests-attempt> Connecting to www.google.com (www.google.com)|142.251.42.132|:80... connected.
my-wget-tests-attempt> HTTP request sent, awaiting response... 200 OK
my-wget-tests-attempt> Length: unspecified [text/html]
my-wget-tests-attempt> Saving to: 'index.html'
my-wget-tests-attempt> index.html [ <=> ] 18.90K --.-KB/s in 0.02s
my-wget-tests-attempt> 2023-07-22 07:11:29 (762 KB/s) - 'index.html' saved [19355]
Just to make sure, let's change the my-wget
derivation again so that it is broken:
final: prev: {
my-wget =
final.wget.overrideAttrs {
pname = "my-wget";
postInstall = ''
rm $out/bin/wget
'';
};
}
Let's confirm in the repl that the hash my-wget
, used in the name of the test, has changed:
$ nix repl
nix-repl> :lf ./.
nix-repl> packages.x86_64-linux.my-wget-tests-attempt-3-fod-special-name.name
"my-wget-tests-attempt-3-fod-special-name-0efdb406714d5f4f279d23c888870f1b"
Great, it has changed! Now when we try to rebuild the test, it should actually be rebuilt:
$ nix build -L .#my-wget-tests-attempt-3-fod-special-name
my-wget-tests-attempt> /nix/store/fzb9wy1yz0hn69vxw12954szvrjnjjgk-stdenv-linux/setup: line 1559: wget: command not found
error: builder for '/nix/store/4vsk6n3lyq6q5l9rrv3d8zyq6piklq4g-my-wget-tests-attempt-3-fod-special-name-0efdb406714d5f4f279d23c888870f1b.drv' failed with exit code 127;
Success! Err, failure... but that's what we were hoping for!
So this seems like it is working well, and doing what we expect. This example is the heart of a fixed-output derivation for operating network-utilizing tests (FODNUT):
What is a FODONUT?
A FODONUT is a fixed-output derivation with a special
name
. Take a hash of the derivation containing the binary you want to test, and put it in thename
of the FOD. You'll be able to access the network in the tests, and the derivation will be correctly rebuilt whenever the main dependency changes.
There is, however, still one problem here5.
Imagine we want to change the test derivation itself. Right now it is downloading an HTML file from http://www.google.com, but we want to change it to instead download an HTML file from http://www.example.com. Let's make that change:
{ my-wget, lib, stdenv }:
let
my-wget-hash = builtins.hashString "md5" my-wget.outPath;
in
stdenv.mkDerivation {
name = "my-wget-tests-attempt-3-fod-special-name-${my-wget-hash}";
# This is the sha256 hash for the string "success", which is output upon this
# test succeeding.
outputHash = "sha256-gbK9TqmMjbZlVPvI12N6GmmhMPMx/rcyt1yqtMSGj9U=";
outputHashMode = "flat";
outputHashAlgo = "sha256";
nativeBuildInputs = [ my-wget ];
# Needed for people using Nix behind a proxy.
impureEnvVars = lib.fetchers.proxyImpureEnvVars;
buildCommand = ''
wget http://www.example.com -O index.html
echo success > "$out"
'';
}
And try rebuilding the test:
$ nix build -L .#my-wget-tests-attempt-3-fod-special-name
Hmm, it doesn't rebuild. This is because we changed the test derivation itself, but we haven't changed the my-wget
derivation at all. Again, Nix sees that it has already built a FOD with this name
and hash
, so doesn't try to rebuild.
The solution to this problem leads us to a full FODONUT.
Attempt 4: A Full FODONUT
A straightforward solution to this is to just hash the full test script, and stick it into the name
of the test derivation. Here's an example of what this might look like:
{ my-wget, lib, stdenv }:
let
buildCommand = ''
wget http://www.google.com -O index.html
echo success > "$out"
'';
value-to-hash = buildCommand + my-wget.outPath;
test-name-hash = builtins.hashString "md5" value-to-hash;
in
stdenv.mkDerivation {
name = "my-wget-tests-attempt-4-fodonut-${test-name-hash}";
# This is the sha256 hash for the string "success", which is output upon this
# test succeeding.
outputHash = "sha256-gbK9TqmMjbZlVPvI12N6GmmhMPMx/rcyt1yqtMSGj9U=";
outputHashMode = "flat";
outputHashAlgo = "sha256";
nativeBuildInputs = [ my-wget ];
# Needed for people using Nix behind a proxy.
impureEnvVars = lib.fetchers.proxyImpureEnvVars;
inherit buildCommand;
}
Here, you can see that we are hashing both the full buildCommand
, as well as the output path of my-wget
. Let's confirm this builds:
$ nix build -L .#my-wget-tests-attempt-4-fodonut
my-wget-tests-attempt> --2023-07-22 08:20:23-- http://www.google.com/
my-wget-tests-attempt> Resolving www.google.com (www.google.com)... 216.58.220.100, 2404:6800:4004:80a::2004
my-wget-tests-attempt> Connecting to www.google.com (www.google.com)|216.58.220.100|:80... connected.
my-wget-tests-attempt> HTTP request sent, awaiting response... 200 OK
my-wget-tests-attempt> Length: unspecified [text/html]
my-wget-tests-attempt> Saving to: 'index.html'
my-wget-tests-attempt> index.html [ <=> ] 18.88K --.-KB/s in 0.03s
my-wget-tests-attempt> 2023-07-22 08:20:23 (730 KB/s) - 'index.html' saved [19334]
Go back into the derivation and try changing the URL from http://www.google.com to http://www.example.com, and rebuilding:
$ nix build -L .#my-wget-tests-attempt-4-fodonut
my-wget-tests-attempt> --2023-07-22 08:24:36-- http://www.example.com/
my-wget-tests-attempt> Resolving www.example.com (www.example.com)... 93.184.216.34, 2606:2800:220:1:248:1893:25c8:1946
my-wget-tests-attempt> Connecting to www.example.com (www.example.com)|93.184.216.34|:80... connected.
my-wget-tests-attempt> HTTP request sent, awaiting response... 200 OK
my-wget-tests-attempt> Length: 1256 (1.2K) [text/html]
my-wget-tests-attempt> Saving to: 'index.html'
my-wget-tests-attempt> index.html 100%[===================>] 1.23K --.-KB/s in 0s
my-wget-tests-attempt> 2023-07-22 08:24:37 (179 MB/s) - 'index.html' saved [1256/1256]
It is rebuilt! Pefect!
Key Takeaways for FODONUTs
So what really are fixed-output derivation for operating network-utilizing tests (FODONUTs)?
FODONUTs are the following:
Fixed-output derivations that output a simple, known string, like "success".
The derivation runs a test that requires access to the internet (so it won't build correctly in a normal sandboxed derivation).
FODONUTs have a hash in their
name
. The hash is carefully crafted to make sure rebuilds happen as a user might expect.
FODONUTs are quite useful for tests, but have a few downsides as well:
They can really only output one bit of information: whether or not the tests succeed. It is not possible for the tests to output some sort of "test report", since FODs aren't able to output anything other than a previously known hashed value6.
Once a test succeeds, it won't run again until its
name
orhash
changes. While this is exactly what we want in many scenarios, there are certain kinds of tests where the exact state of the data we are accessing on the internet has a big affect on the outcome of the test. Data from the internet may change frequently, and cause frequent test failures.For instance, with the
my-wget
example, we are making surewget
can download the front page of Google. In this case, Google is quite reliable. Ifwget
is not able to download the front page of Google, is is likely a problem withwget
, not Google.7As a rule of thumb, FODONUTs only make sense for tests that download data that isn't going to frequently change. Or, at least, data that won't change in a way that causes the tests to fail. If you're using a FODONUT, and your tests frequently fail because of changes to data from the internet, you should rethink whether FODONUTs are right for your use-case8.
Sometimes you have to be quite careful exactly what information ends up in the the
name
of a FODONUT. You want to make sure this causes the derivation to be rebuilt only when it should be rebuilt. You don't want too many rebuilds, and you don't want your tests to accidentally not get run when they should.
Examples of FODONUTs in the Wild
I know of a few examples of FODONUTs in the wild:
Stack / Hpack version test9 in Nixpkgs
This is a test to make sure that the version of the
hpack
dependency used bystack
in Nixpkgs matches the same version as upstream. This test downloads a statically-linkedstack
binary from GitHub, and compares versions of thehpack
dependency with thestack
from Nixpkgs.There are two interesting things to notice here:
The
name
of the test derivation contains thestack
andhpack
version numbers, so if thestack
orhpack
version is updated in Nixpkgs, this test derivation will be rebuilt.The Haskell infrastructure in Nixpkgs is setup so that
stack
andhpack
may be updated semi-automatically, so this test derivation will helpfully catch whenstack
orhpack
are updated to versions that are incompatible with upstream releases.You can see that the test script uses the
cacert
derivation. However, changes in thecacert
derivation are not likely to cause this test to start failing, so thecacert
derivation is explicitly not included anywhere in the hash added to the testname
. We have to be somewhat careful with how we write the test script.
You may be wondering why this couldn't just be split into a FOD that downloads the upstream statically-linked
stack
binary, and a separate, normal derivation that performs the test.It could be split-up like that, but then we'd have to manually bump the hash of the upstream
stack
binary. Writing this as a FODONUT saves us the trouble of having to manually update the hash.The tests for the fetchers in Nixpkgs, like
fetchurl
.While these aren't exactly FODONUTs, they do share some of the same tactics.
-
This doesn't technically use FODONUTs, but it does contain a FOD with a carefully constructed
name
. The repro-get pkg-version test in Nixpkgs
This was authored by Matthew Croughan and Robert Hensing.
I imagine there are more examples of FODONUTs being used in other projects and proprietary repos. If you have an example of a FODONUT, please share!
Conclusion
While I'm sure I'm not the first to come up with the idea of using FODs to run tests that access the internet, I haven't seen any other blog posts talking about the idea. This post is an attempt to write up an easy-to-understand explanation of how FODONUTS work.
FODONUTs are quite a helpful technique for running tests within Nix derivations. When building software with Nix, it is very common to want to get data from the internet to use in tests, but it is often too difficult or annoying to download that data with a fixed-output derivation. FODONUTs can be a nice compromise in this situation.
Footnotes
I read this as "foe donuts", or "faux donuts".↩︎
Another common solution is to standup a webserver like Nginx on localhost within a normal Nix derivation, and make requests to it with
wget
. The advantage with this is that everything happens within a normal Nix derivation, and in theory it is more reproducible than accessing something out on the internet. The disadvantage is that you must download all resources required for the tests using FODs. While this is possible for some types of tests, for others it is quite a big limitation.↩︎In theory, if you've already output one thing with a given name/hash, it doesn't matter if you change the build steps, you'll still get the same value if you rebuild it.
In practice, I'm not sure how good of a decision this is, since it can be quite surprising to users when they update the build steps in an FOD, but forget to change the hash. But it is also nice in some ways.↩︎
You can also work around this by disabling the build
sandbox
:$ nix build -L .#my-wget-tests-attempt-1-normal-derivation --option sandbox false
But this requires you to run the build as a
trusted-user
, which is quite limiting, and not something that is allowed in Nixpkgs.↩︎There are a couple other things that we could talk about as well, which aren't important enough for the main post:
Is it safe to use an MD5 hash here?
Yes. There are no security implications here, so MD5 is alright. It is extremely unlikely we get a collision, and it is really not the end of the world if there just happens to be a collision.
Is there a better method than hashing the full path of the
my-wget
output?Yes. We could just take the hash from the path of the
my-wget
output, and use that. For example, the full path of themy-wget
output is:$ nix path-info .#my-wget /nix/store/qn523zxch23jsa5jpc4b2fhr25zh85kn-my-wget-1.21.4
Instead of MD5 hashing this path, we could just directly use the hash
qn523zxch23jsa5jpc4b2fhr25zh85kn
. The big advantage of this is that we don't have to compute an MD5 hash again, and for people that use an alternative/nix/store
path, they will get the same value.There really isn't any downside here, except it is slightly more difficult to code up and explain.
Keep in mind that Nix builds have a concept of a "build log". It is possible to output test failures to the build log, and inspect them at a later time.
For instance, try changing the
my-wget-tests-attempt-4-fodonut
derivation to download from a domainlalalathisdomaindoesnotexist.com
instead ofgoogle.com
. Here's what happens when you run this test:$ nix build -L .#my-wget-tests-attempt-4-fodonut warning: Git tree '/home/illabout/git/example-fodonuts' is dirty my-wget-tests-attempt> --2023-07-22 11:30:59-- http://www.lalalathisdomaindoesnotexist.com/ my-wget-tests-attempt> Resolving www.lalalathisdomaindoesnotexist.com (www.lalalathisdomaindoesnotexist.com)... failed: Name or service not known. my-wget-tests-attempt> wget: unable to resolve host address 'www.lalalathisdomaindoesnotexist.com' error: builder for '/nix/store/inw2snkn1cl7v6x83x0z03006q4rn8q7-my-wget-tests-attempt-4-fodonut-6c62f59d6308b771c9a6353f23f518e0.drv' failed with exit code 4;
We can of course see the build failure here, and Nix also has a built-in way of viewing the build logs for this build failure:
↩︎$ nix log /nix/store/inw2snkn1cl7v6x83x0z03006q4rn8q7-my-wget-tests-attempt-4-fodonut-6c62f59d6308b771c9a6353f23f518e0.drv --2023-07-22 11:30:59-- http://www.lalalathisdomaindoesnotexist.com/ Resolving www.lalalathisdomaindoesnotexist.com (www.lalalathisdomaindoesnotexist.com)... failed: Name or service not known. wget: unable to resolve host address 'www.lalalathisdomaindoesnotexist.com'
In practice, this likely depends on things like the stability of your internet, what sort of data you are downloading and using in your tests, etc.↩︎
Another way to think about this: if your FODONUTs frequently fail because of changes to data from the internet, you likely want tests that are able to run more frequently than just whenever the underlying derivation changes.
In our
my-wget
example, in practice, we might be really worried about the compatibility betweenmy-wget
and the upstream website we're trying to download from. However, themy-wget
derivation might not change much, so the tests won't actually be run that often. In this case, a FODONUT might not be the best choice, and you should really try to find some way of running tests outside of a Nix derivation.If, for some reason, you don't have any way to run tests except as Nix derivations, you might be able to force rebuilds of your FODONUT by using things like
builtins.currentTime
in thename
. Or possibly just turning off Nix's build-time sandox.↩︎I originally wrote this test.↩︎
tags: nixos