2019-03-16

I have created a Nix derivation for a Jupyter environment for the fast.ai Practical Deep Learning for Coders v3 course. This provides a Python environment from within Jupyter with the necessary libraries to go through the fast.ai course, including fastai, pytorch, numpy, matplotlib, etc. It is also setup to use CUDA.

I won't reproduce the Nix derivation here, but you can find it on the following link:

https://github.com/cdepillabout/course-v3/blob/6ecf90f640f327dc05b167f45fb7ba88bc265281/default.nix

I recommend you clone the fast.ai course repo, and then copy the default.nix file above into that directory. From there, you can run nix-build¹.

nix-build will produce a file called result/bin/run-fastai-jupyter. This is a shell script you can run to start Jupyter. Once you've run this shell script, you can access Jupyter at http://localhost:8888/.

This Nix derivation is based on the fast.ai course repo at commit 7ed77f47 from 2019-03-01. It may or may not run with later versions of the fast.ai course repo.

CUDA

I've tested the above derivation on an Ubuntu-18.04 system with an NVIDIA driver version 396 (which uses CUDA-9.2). It should work with other versions of the NVIDIA drivers and CUDA, but some things may have to be changed in the Nix file.

The Nix file takes an argument specifying the directory where the NVIDIA and CUDA libraries are located. You can specify this as follows:

$ nix-build --argstr nvidiaLibsPath /usr/lib/x86_64-linux-gnu

In the /usr/lib/x86_64-linux-gnu directory, you should be able to find the libraries like libcuda.so, libnvidia-fatbinaryloader.so.*, libnvidia-ptxjitcompiler.so, etc. If these libraries exist at a different path on your system, you will have to specify this path when running nix-build like above.

In the run-fastai-jupyter script, these necessary libraries are loaded into the Python process with LD_PRELOAD. This hack is necessary because of the way Nix and CUDA interact. The CUDA libraries are tied to the version of the NVIDIA driver for your video card. However, Nix wants to control all the versions of the libraries used, so there is no way to tell Nix to just use the existing libraries on your system. This is quite unfortunate, but appears to be necessary to get everything working together.

Related Work

There is also a repository from Tweag I/O that enables an easy way to setup Jupyter. I haven't used this, so I can't compare it to my Nix derivation from above, but you may be interested in checking it out.

Conclusion

This Nix derivation should give you a working Jupyter installation that you can use to go through the fast.ai course on deep learning.

The way it works with CUDA is slightly hacky, but everything else should be fully reproducible.

Footnotes

tags: machine learning, nixos