NVIDIA provides an optimized GPU library for deep neural network primitives called cuDNN. These primitives include operations such as forward propagation, convolutions, back propagation, activation functions (such as sigmoid, ReLU, and tanh), and gradient descent. cuDNN is what most of the mainstream deep neural network frameworks such as Tensorflow use as a backend for NVIDIA GPUs. This is provided for free by NVIDIA , but has to be downloaded separately from the CUDA Toolkit.
More information on cuDNN is available here: https://developer.nvidia.com/cudnn.
