Tensorflow GPU Checks

Once we have, think we have, suspect we have, hope we have or are otherwise inclined to think we may have tensorflow GPU support, it's time to check. The below are snippets that may help us check:

# One git issue indicated this may be required. I lost the issue ID

import os 

os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID" # see issue #152
os.environ["CUDA_VISIBLE_DEVICES"]="0" 

# import tensorflow
import tensorflow as tf

# Check no.1
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))

# Check no.2
with tf.device('/gpu:0'):
    a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
    b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
    c = tf.matmul(a, b)

with tf.Session() as sess:
    print (sess.run(c))

# Check no.3
# Built in - this should probably be check no. 1
tf.test.is_gpu_available(
    cuda_only=False,
    min_cuda_compute_capability=None
)

Results I get are:
Check no.1
Device mapping:
/job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device
2019-03-28 20:54:39.026230: I tensorflow/core/common_runtime/direct_session.cc:316] Device mapping:
/job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device

Hmm, CPU not GPU. Fail. We expect all of them to fail, but they are vaguely interesting.

Check no.2
<long stack trace>
InvalidArgumentError: Cannot assign a device for operation MatMul: node MatMul (defined at <ipython-input-7-5d3b23a68111>:4) was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0 ]. Make sure the device specification refers to a valid device.
[[MatMul]]

Errors may have originated from an input operation.
Input Source operations connected to node MatMul:
 a (defined at <ipython-input-7-5d3b23a68111>:2)
 b (defined at <ipython-input-7-5d3b23a68111>:3)


Check no.3
False

So - for me at least, tensorflow isn't using the GPU. Pah!

I actually already knew this, as i get a driver mismatch error. This is just here in case it's useful.


Added Later
Once my home built tensorflow gpu libs are built and up and running this is what i get, if you wish to compare:

import os 

os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID" # see issue #152 
os.environ["CUDA_VISIBLE_DEVICES"]="


import tensorflow as tf

sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))

with tf.device('/gpu:0'):
    a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
    b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
    c = tf.matmul(a, b)

with tf.Session() as sess:
    print (sess.run(c))

tf.test.is_gpu_available(
    cuda_only=False,
    min_cuda_compute_capability=None
)

Returns:

Device mapping:
/job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device
/job:localhost/replica:0/task:0/device:XLA_GPU:0 -> device: XLA_GPU device
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1
[[22. 28.]
[49. 64.]]

2019-03-29 12:28:30.622272: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1674] Adding visible gpu devices: 0
2019-03-29 12:28:30.622303: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1082] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-03-29 12:28:30.622308: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1088] 0
2019-03-29 12:28:30.622311: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1101] 0: N
2019-03-29 12:28:30.622403: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1222] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6722 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
2019-03-29 12:28:30.622532: I tensorflow/core/common_runtime/direct_session.cc:316] Device mapping:
/job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device
/job:localhost/replica:0/task:0/device:XLA_GPU:0 -> device: XLA_GPU device
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1

2019-03-29 12:28:30.625908: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1674] Adding visible gpu devices: 0
2019-03-29 12:28:30.625938: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1082] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-03-29 12:28:30.625943: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1088] 0
2019-03-29 12:28:30.625946: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1101] 0: N
2019-03-29 12:28:30.626034: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1222] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6722 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
2019-03-29 12:28:30.645547: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1674] Adding visible gpu devices: 0
2019-03-29 12:28:30.645575: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1082] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-03-29 12:28:30.645579: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1088] 0
2019-03-29 12:28:30.645582: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1101] 0: N
2019-03-29 12:28:30.645645: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1222] Created TensorFlow device (/device:GPU:0 with 6722 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
Out[10]: True


Bolding added just to highlight what I was looking for. While I'm in a musical mood - it turns out I have found what I was looking for.