O'Reilly logo

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Neural Network Programming with TensorFlow

Book Description

Neural Networks and their implementation decoded with TensorFlow

About This Book

  • Develop a strong background in neural network programming from scratch, using the popular Tensorflow library.
  • Use Tensorflow to implement different kinds of neural networks – from simple feedforward neural networks to multilayered perceptrons, CNNs, RNNs and more.
  • A highly practical guide including real-world datasets and use-cases to simplify your understanding of neural networks and their implementation.

Who This Book Is For

This book is meant for developers with a statistical background who want to work with neural networks. Though we will be using TensorFlow as the underlying library for neural networks, book can be used as a generic resource to bridge the gap between the math and the implementation of deep learning. If you have some understanding of Tensorflow and Python and want to learn what happens at a level lower than the plain API syntax, this book is for you.

What You Will Learn

  • Learn Linear Algebra and mathematics behind neural network.
  • Dive deep into Neural networks from the basic to advanced concepts like CNN, RNN Deep Belief Networks, Deep Feedforward Networks.
  • Explore Optimization techniques for solving problems like Local minima, Global minima, Saddle points
  • Learn through real world examples like Sentiment Analysis.
  • Train different types of generative models and explore autoencoders.
  • Explore TensorFlow as an example of deep learning implementation.

In Detail

If you're aware of the buzz surrounding the terms such as "machine learning," "artificial intelligence," or "deep learning," you might know what neural networks are. Ever wondered how they help in solving complex computational problem efficiently, or how to train efficient neural networks? This book will teach you just that.

You will start by getting a quick overview of the popular TensorFlow library and how it is used to train different neural networks. You will get a thorough understanding of the fundamentals and basic math for neural networks and why TensorFlow is a popular choice Then, you will proceed to implement a simple feed forward neural network. Next you will master optimization techniques and algorithms for neural networks using TensorFlow. Further, you will learn to implement some more complex types of neural networks such as convolutional neural networks, recurrent neural networks, and Deep Belief Networks. In the course of the book, you will be working on real-world datasets to get a hands-on understanding of neural network programming. You will also get to train generative models and will learn the applications of autoencoders.

By the end of this book, you will have a fair understanding of how you can leverage the power of TensorFlow to train neural networks of varying complexities, without any hassle. While you are learning about various neural network implementations you will learn the underlying mathematics and linear algebra and how they map to the appropriate TensorFlow constructs.

Style and Approach

This book is designed to give you just the right number of concepts to back up the examples. With real-world use cases and problems solved, this book is a handy guide for you. Each concept is backed by a generic and real-world problem, followed by a variation, making you independent and able to solve any problem with neural networks. All of the content is demystified by a simple and straightforward approach.

Downloading the example code for this book. You can download the example code files for all Packt books you have purchased from your account at http://www.PacktPub.com. If you purchased this book elsewhere, you can visit http://www.PacktPub.com/support and register to have the code file.

Table of Contents

  1. Preface
    1. What this book covers
    2. What you need for this book
    3. Who this book is for
    4. Conventions
    5. Reader feedback
    6. Customer support
      1. Downloading the example code
      2. Downloading the color images of this book
      3. Errata
      4. Piracy
      5. Questions
  2. Maths for Neural Networks
    1. Understanding linear algebra
      1. Environment setup
        1. Setting up the Python environment in Pycharm
      2. Linear algebra structures
        1. Scalars, vectors, and matrices
        2. Tensors
      3. Operations
        1. Vectors
        2. Matrices
        3. Matrix multiplication
        4. Trace operator
        5. Matrix transpose
        6. Matrix diagonals
        7. Identity matrix
        8. Inverse matrix
      4. Solving linear equations
      5. Singular value decomposition
      6. Eigenvalue decomposition
      7. Principal Component Analysis
    2. Calculus
      1. Gradient
      2. Hessian
      3. Determinant
    3. Optimization
      1. Optimizers
    4. Summary
  3. Deep Feedforward Networks
    1. Defining feedforward networks
    2. Understanding backpropagation
    3. Implementing feedforward networks with TensorFlow
    4. Analyzing the Iris dataset
      1. Code execution
    5. Implementing feedforward networks with images
      1. Analyzing the effect of activation functions on the feedforward networks accuracy
    6. Summary
  4. Optimization for Neural Networks
    1. What is optimization?
    2. Types of optimizers
    3. Gradient descent
      1. Different variants of gradient descent
      2. Algorithms to optimize gradient descent
    4. Which optimizer to choose
      1. Optimization with an example
    5. Summary
  5. Convolutional Neural Networks
    1. An overview and the intuition of CNN
      1. Single Conv Layer Computation
      2. CNN in TensorFlow
        1. Image loading in TensorFlow
    2. Convolution operations
      1. Convolution on an image
      2. Strides
    3. Pooling
      1. Max pool
      2. Example code
        1. Average pool
    4. Image classification with convolutional networks
      1. Defining a tensor for input images and the first convolution layer
        1. Input tensor
        2. First convolution layer
        3. Second convolution layer
        4. Third convolution layer
        5. Flatten the layer
        6. Fully connected layers
        7. Defining cost and optimizer
        8. Optimizer
        9. First epoch
        10. Plotting filters and their effects on an image
    5. Summary
  6. Recurrent Neural Networks
    1. Introduction to RNNs
      1. RNN implementation
        1. Computational graph
      2. RNN implementation with TensorFlow
        1. Computational graph
    2. Introduction to long short term memory networks
      1. Life cycle of LSTM
      2. LSTM implementation
        1. Computational graph
    3. Sentiment analysis
      1. Word embeddings
      2. Sentiment analysis with an RNN
        1. Computational graph
    4. Summary
  7. Generative Models
    1. Generative models
      1. Discriminative versus generative models
      2. Types of generative models
        1. Autoencoders
        2. GAN
        3. Sequence models
    2. GANs
      1. GAN with an example
      2. Types of GANs
        1. Vanilla GAN
        2. Conditional GAN
        3. Info GAN
        4. Wasserstein GAN
        5. Coupled GAN
    3. Summary
  8. Deep Belief Networking
    1. Understanding deep belief networks
      1. DBN implementation
        1. Class initialization
        2. RBM class
          1. Pretraining the DBN
    2. Model training
    3. Predicting the label
    4. Finding the accuracy of the model
    5. DBN implementation for the MNIST dataset
      1. Loading the dataset
      2. Input parameters for a DBN with 256-Neuron RBM layers
      3. Output for a DBN with 256-neuron RBN layers
    6. Effect of the number of neurons in an RBM layer in a DBN
      1. An RBM layer with 512 neurons
      2. An RBM layer with 128 neurons
      3. Comparing the accuracy metrics
    7. DBNs with two RBM layers
    8. Classifying the NotMNIST dataset with a DBN
    9. Summary
  9. Autoencoders
    1. Autoencoder algorithms
    2. Under-complete autoencoders
    3. Dataset
    4. Basic autoencoders
      1. Autoencoder initialization
      2. AutoEncoder class
      3. Basic autoencoders with MNIST data
        1. Basic autoencoder plot of weights
        2. Basic autoencoder recreated images plot
      4. Basic autoencoder full code listing
      5. Basic autoencoder summary
    5. Additive Gaussian Noise autoencoder
      1. Autoencoder class
      2. Additive Gaussian Autoencoder with the MNIST dataset
        1. Training the model
        2. Plotting the weights
      3. Plotting the reconstructed images
      4. Additive Gaussian autoencoder full code listing
      5. Comparing basic encoder costs with the Additive Gaussian Noise autoencoder
      6. Additive Gaussian Noise autoencoder summary
    6. Sparse autoencoder
      1. KL divergence
        1. KL divergence in TensorFlow
          1. Cost of a sparse autoencoder based on KL Divergence
      2. Complete code listing of the sparse autoencoder
      3. Sparse autoencoder on MNIST data
      4. Comparing the Sparse encoder with the Additive Gaussian Noise encoder
    7. Summary
  10. Research in Neural Networks
    1. Avoiding overfitting in neural networks
      1. Problem statement
      2. Solution
      3. Results
    2. Large-scale video processing with neural networks
      1. Resolution improvements
      2. Feature histogram baselines
      3. Quantitative results
    3. Named entity recognition using a twisted neural network
      1. Example of a named entity recognition
      2. Defining Twinet
      3. Results
    4. Bidirectional RNNs
      1. BRNN on TIMIT dataset
    5. Summary
  11. Getting started with TensorFlow
    1. Environment setup
    2. TensorFlow comparison with Numpy
    3. Computational graph
      1. Graph
      2. Session objects
      3. Variables
      4. Scope
      5. Data input
      6. Placeholders and feed dictionaries
    4. Auto differentiation
    5. TensorBoard