While on most occasions simple
pip install tensorflow works just fine, certain combinations of hardware may be incompatible with the repository-installed tensorflow package. In this brief tutorial I will build the latest tensorflow 2.3.1 python package from the source. This tutorial may also be helpfull for those who want to update to the latest tensorflow version on older GPUs because older hardware support was removed from the precompiled version since 2.3.0.
Prepare the building environment
Obtain the following docker container:
docker pull tensorflow/tensorflow:devel-gpu
Choose a place and create a directory that you will share with the container. In my case, I will use
/home/alexandr/temp/tensorflow. Then enter the working directory and start the docker container
cd /home/alexandr/temp/tensorflow docker run -it -w /tensorflow_src -v $(pwd):/share tensorflow/tensorflow:devel-gpu bash
Update the repository within the container and chose the latest stable branch (at the time of writing that was 2.3)
git pull git checkout r2.3
Next, upgrade pip and install few python dependencies
/usr/bin/python3 -m pip install --upgrade pip pip3 install six numpy wheel keras_applications keras_preprocessing
Figure out the CPU limitations on the target machine
You need to tell the compiler what instructions to avoid in the final binaries. This is not an obvious step, as those limitations are machine specific. You may want to consult with internet and even to use some trial-and-error to see what flags are required for making the binaries stable on your particular machine.
If you are compiling on the target machine,
-march=native should suffice, as it should enable all instructions your CPU supports. If you crosscompile, as I am in this example, then you need to dig deeper.
One approach is to look at
grep flags /proc/cpuinfo | head -n 1 which shows CPU features. In my case, I have a laptop where latest tensorflow works out of the box and a desktop PC with GPU where it does not. Comparing the lists from both machines, I find that the desktop PC lacks the following items:
'ida', 'bmi2', 'smep', 'rtm', 'bmi1', 'fma', 'f16c', 'hle', 'avx2', 'smx', 'adx', 'avx', 'mpx'.
Here we can see that
ida stands for Intel Dynamic Acceleration and is a part of CPU’s Thermal and Power Management, so it is not very likely to be the breaking factor in my case. In the same vein,
smep, rtm, hle, smx, and
mpx features are unlikely to affect tensorflow execution. I could not easily find wheter
adx are used by tensorflow binaries.
On the other hand
avx2 (Advanced Vector Extensions),
bmi1, bmi2 (1st/group bit manipulation extensions), and
fma (Fused multiply-add) seem to be quite important for tensorflow. Therefore, I will use the following combination of flags to build tensorflow binaries
-march=native -mno-avx -mno-avx2 -mno-fma -mno-bmi -mno-bmi2
Configure a tensorflow’s build chain
In the docker container execute
to start a configuration manager. For most questions you can just select the default answer.
One important question is about compute capability of your GPU. Since the default option may not include your GPU type, its is better to check it ahead of time here and input into the field provided. If you misspecify your GPU compute capability, then you are likely to get the following error message at an attempt to use CUDA in tensorflow:
InternalError: CUDA runtime implicit initialization on GPU:0 failed. Status: device kernel image is invalid
When you get to the question about optimization flags, you should input the flags we came up with in the previous section. In my case I also added a
-Wno-sign-compare flag. Once you are done, you can start the building process by running
bazel build //tensorflow/tools/pip_package:build_pip_package --local_ram_resources=16384
If bazel complains about its version it will also likely provide you with a one-liner to update it.
Build process requires a substantial amount of RAM especially on machines with many cores, so you may want to limit RAM usage by using flag
--local_ram_resources=16384. In my case I limit it to 16 GiB out of 24 GiB available on the machine. Another way of limiting resources could be restricting the number of threads to a smaller number using flag
The build will take quite a long time.
Prepare and install a python package
Execute the following command to assemble a python package
Now you should be able to see a
.whl file in the mounted directory of the host machine. Copy that file to the target machine and then install with
pip. The filename suggests which python version you should use at the target machine. If you have a different version, you can use conda’s enviroments to create a separate environment for tensorflow with a required version of python.
conda create -n "tensorflow2" python=3.6 conda activate tensorflow2 pip install tensorflow-2.3.1-cp36-cp36m-linux_x86_64.whl
Now you should be able to import and use tensorflow in this new environment.
That is it!