Ubuntu 16.04 LTS with NVIDIA CUDA Docker

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Patrick

Patrick

Administrator
Staff member
Dec 21, 2010
12,513
5,802
113
Thank you to @Patriot

Code: 
$ sudo nvidia-docker run --rm nvidia/cuda nvidia-smi
Using default tag: latest
latest: Pulling from nvidia/cuda
ba76e97bb96c: Pull complete
4d6181e6b423: Pull complete
4854897be9ac: Pull complete
4458f3097eef: Pull complete
9989a8de1a9e: Pull complete
97b9fecc40a9: Pull complete
49174734b90f: Pull complete
380d7a401db4: Pull complete
214581faccc5: Pull complete
e767bb63d75c: Pull complete
Digest: sha256:f0f82a1dd6c3cb516af42553e70db16c10769a8ef1f18597e8679c35e73efa7e
Status: Downloaded newer image for nvidia/cuda:latest
Fri Nov 18 23:49:04 2016
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.20                 Driver Version: 375.20                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GRID M40            Off  | 0000:07:00.0     Off |                  N/A |
| 38%   22C    P8     8W /  53W |    265MiB /  4073MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GRID M40            Off  | 0000:08:00.0     Off |                  N/A |
| 38%   23C    P8     8W /  53W |    264MiB /  4073MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GRID M40            Off  | 0000:09:00.0     Off |                  N/A |
| 38%   26C    P8     8W /  53W |    264MiB /  4073MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GRID M40            Off  | 0000:0A:00.0     Off |                  N/A |
| 38%   26C    P8     8W /  53W |    264MiB /  4073MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   4  GRID M40            Off  | 0000:83:00.0     Off |                  N/A |
| 38%   23C    P8     8W /  53W |    264MiB /  4073MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   5  GRID M40            Off  | 0000:84:00.0     Off |                  N/A |
| 38%   23C    P8     8W /  53W |    264MiB /  4073MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   6  GRID M40            Off  | 0000:85:00.0     Off |                  N/A |
| 38%   24C    P8     8W /  53W |    264MiB /  4073MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   7  GRID M40            Off  | 0000:86:00.0     Off |                  N/A |
| 38%   24C    P8     8W /  53W |    264MiB /  4073MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
 
  • Like
Reactions: gigatexal
Patrick

Patrick

Administrator
Staff member
Dec 21, 2010
12,513
5,802
113
Oh this is really cool now! Here is the same server 2 minutes later running CUDA 7.5 only on GPU 0 and CUDA 8 on GPU 1 in different containers. Each is only seeing 1 GPU. No need for VT-d.

Code: 
$ sudo NV_GPU=0 nvidia-docker run -ti nvidia/cuda:7.5-devel
7.5-devel: Pulling from nvidia/cuda
ba76e97bb96c: Already exists
4d6181e6b423: Already exists
4854897be9ac: Already exists
4458f3097eef: Already exists
9989a8de1a9e: Already exists
97b9fecc40a9: Already exists
7553cfc5e743: Pull complete
cacbaf267bfa: Pull complete
633e39e1386c: Pull complete
6dfe826be980: Pull complete
8a343e9aec87: Pull complete
Digest: sha256:57de80dd6197dfd5e9489cbe4b0fd356e29d7e3ab0a03688570e9f49a1c74d19
Status: Downloaded newer image for nvidia/cuda:7.5-devel
root@43e4a2318b63:/# nvidia-smi
Fri Nov 18 23:55:22 2016
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.20                 Driver Version: 375.20                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GRID M40            Off  | 0000:07:00.0     Off |                  N/A |
| 38%   22C    P8     8W /  53W |    265MiB /  4073MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
root@43e4a2318b63:/# ^C
root@43e4a2318b63:/# ubuntu@8xgpublade01:~$ docker ps
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
ubuntu@8xgpublade01:~$ sudo docker ps
CONTAINER ID        IMAGE                   COMMAND             CREATED             STATUS              PORTS               NAMES
43e4a2318b63        nvidia/cuda:7.5-devel   "/bin/bash"         3 minutes ago       Up 3 minutes                            condescend                                               ing_blackwell
ubuntu@8xgpublade01:~$ sudo NV_GPU=1 nvidia-docker run -ti nvidia/cuda
root@d963b9be8586:/# nvidia-smi
Fri Nov 18 23:58:41 2016
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.20                 Driver Version: 375.20                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GRID M40            Off  | 0000:08:00.0     Off |                  N/A |
| 38%   22C    P8     8W /  53W |    264MiB /  4073MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

ubuntu@8xgpublade01:~$ sudo docker ps
CONTAINER ID        IMAGE                   COMMAND             CREATED             STATUS              PORTS               NAMES
d963b9be8586        nvidia/cuda             "/bin/bash"         32 seconds ago      Up 32 seconds                           lonely_knu                                               th
43e4a2318b63        nvidia/cuda:7.5-devel   "/bin/bash"         4 minutes ago       Up 4 minutes                            condescend
 
Patrick

Patrick

Administrator
Staff member
Dec 21, 2010
12,513
5,802
113
Ha! Ok... this is borderline obscene.

I managed to get a Monero GPU miner working in a Docker container mining on GPU 7
upload_2016-11-18_16-42-11.png

This is certainly the wrong way to do the GPU miner. It installs a build environment, wget, unzip and then downloads the source/ compiles, and runs via manual command line) but I have it down to a series of commands now.

One GRID M40 GPU is about a Geforce 750 Ti so it is around 225 H/s at 36w.
 
  • Like
Reactions: gigatexal, whitey, Chuntzu and 2 others
Patrick

Patrick

Administrator
Staff member
Dec 21, 2010
12,513
5,802
113
Being on HI time still (despite being back in CA) and got a working Dockerfile for this:

upload_2016-11-26_1-28-18.png

That is working all from:
Code: 
NV_GPU=1 nvidia-docker run --name GPU1_Monero -ti monerogpu2
I can swap NV_GPU and run on different GPUs. That is a lot of fun as a proof of concept.

Seems like this is a stable configuration as well. ~7 days uptime on the more manual version across 8 GPUs.
 
Last edited:
  • Like
Reactions: Patriot
Patrick

Patrick

Administrator
Staff member
Dec 21, 2010
12,513
5,802
113
Moved to a second box with the CPU and GPU docker images
Code: 
ubuntu@gbuplade3:~/monerocpu$ NV_GPU=1 nvidia-docker run --name GPU1_Monero -d monerogpu2
05c9503c862de1786e96d35b9f919f41ad393dd05c07d3504164e820c7276a6f
ubuntu@gbuplade3:~/monerocpu$ NV_GPU=2 nvidia-docker run --name GPU2_Monero -d monerogpu2
74350d82df1ba67f6930bb7ae7e28120418b32aef2e24e29ca98329061fc6f15
ubuntu@gbuplade3:~/monerocpu$ NV_GPU=3 nvidia-docker run --name GPU3_Monero -d monerogpu2
9333e337430cfd3c774ec765360a056e26939e2a1f81d90feddd351eb30f8259
ubuntu@gbuplade3:~/monerocpu$ docker run --name CPU_Monero -d monerocpu
f77561e2c7f26e6204ed149fb92a4b58e10b453d69b8e83260add2a71484d53c
ubuntu@gbuplade3:~/monerocpu$ nvidia-docker run --rm nvidia/cuda nvidia-smi
Sun Nov 27 19:16:21 2016
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.20                 Driver Version: 375.20                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GRID M40            Off  | 0000:07:00.0     Off |                  N/A |
| 38%   34C    P0    35W /  53W |    943MiB /  4073MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GRID M40            Off  | 0000:08:00.0     Off |                  N/A |
| 38%   37C    P0    38W /  53W |    943MiB /  4073MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GRID M40            Off  | 0000:09:00.0     Off |                  N/A |
| 44%   46C    P0    37W /  53W |    943MiB /  4073MiB |     98%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GRID M40            Off  | 0000:0A:00.0     Off |                  N/A |
| 41%   42C    P0    37W /  53W |    943MiB /  4073MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
BTW @gigatexal if you want to try... I can help get you setup wth CPU/ GPU docker files.
 
  • Like
Reactions: gigatexal
C

capn_pineapple

Active Member
Aug 28, 2013
356
80
28
So... I'm still confused on the whole nVidia Docker thing. Is it for GPU passthrough to an app, or does docker "run" on the 1000's of cores made available on the GPU increasing the amount of containers you can handle on a system?
 
Patrick

Patrick

Administrator
Staff member
Dec 21, 2010
12,513
5,802
113
capn_pineapple said:
So... I'm still confused on the whole nVidia Docker thing. Is it for GPU passthrough to an app, or does docker "run" on the 1000's of cores made available on the GPU increasing the amount of containers you can handle on a system?
Click to expand...
So the key issue you have is that some software frameworks require different versions of CUDA. Having CUDA 7.0, 7.5 and 8.0 on a single system is not always the easiest thing to have work properly.

Using nvidia-docker you can specify an environment with a specific CUDA version. So you can have caffe with CUDA 7.5 run on GPU0, finish a task and 30 seconds later run a CUDA 8.0 with DIGITS in another container on GPU0 (or on GPU1 or GPU0 + GPU1). Very clean and easy to make work. You can create Docker images that rely upon certain CUDA version and then push them out to other machines.

If you have not tried the deep learning stuff yet (especially switching frameworks), all of the dependencies can be painful to manage. Using Docker does help with that a lot.
 
  • Like
Reactions: gigatexal and capn_pineapple
gigatexal

gigatexal

I'm here to learn
Nov 25, 2012
2,913
607
113
Portland, Oregon
alexandarnarayan.com
Patrick said:
Moved to a second box with the CPU and GPU docker images
Code: 
ubuntu@gbuplade3:~/monerocpu$ NV_GPU=1 nvidia-docker run --name GPU1_Monero -d monerogpu2
05c9503c862de1786e96d35b9f919f41ad393dd05c07d3504164e820c7276a6f
ubuntu@gbuplade3:~/monerocpu$ NV_GPU=2 nvidia-docker run --name GPU2_Monero -d monerogpu2
74350d82df1ba67f6930bb7ae7e28120418b32aef2e24e29ca98329061fc6f15
ubuntu@gbuplade3:~/monerocpu$ NV_GPU=3 nvidia-docker run --name GPU3_Monero -d monerogpu2
9333e337430cfd3c774ec765360a056e26939e2a1f81d90feddd351eb30f8259
ubuntu@gbuplade3:~/monerocpu$ docker run --name CPU_Monero -d monerocpu
f77561e2c7f26e6204ed149fb92a4b58e10b453d69b8e83260add2a71484d53c
ubuntu@gbuplade3:~/monerocpu$ nvidia-docker run --rm nvidia/cuda nvidia-smi
Sun Nov 27 19:16:21 2016
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.20                 Driver Version: 375.20                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GRID M40            Off  | 0000:07:00.0     Off |                  N/A |
| 38%   34C    P0    35W /  53W |    943MiB /  4073MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GRID M40            Off  | 0000:08:00.0     Off |                  N/A |
| 38%   37C    P0    38W /  53W |    943MiB /  4073MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GRID M40            Off  | 0000:09:00.0     Off |                  N/A |
| 44%   46C    P0    37W /  53W |    943MiB /  4073MiB |     98%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GRID M40            Off  | 0000:0A:00.0     Off |                  N/A |
| 41%   42C    P0    37W /  53W |    943MiB /  4073MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
BTW @gigatexal if you want to try... I can help get you setup wth CPU/ GPU docker files.
Click to expand...
Of course @Patrick ! It would be nice to see what the single 980ti will do. I'll get ubuntu 16.04 on the box and let's get this going.
 
Patrick

Patrick

Administrator
Staff member
Dec 21, 2010
12,513
5,802
113
gigatexal said:
Of course @Patrick ! It would be nice to see what the single 980ti will do. I'll get ubuntu 16.04 on the box and let's get this going.
Click to expand...
Just sent you fairly huge instructions via conversation and included some fixes for issues I have seen while playing with a few different setups.
 
  • Like
Reactions: gigatexal
You must log in or register to reply here.
Top Bottom