Deep Learning Quick Reference

Deep Learning Quick Reference

by Mike Bernico
Released March 2018
Publisher(s): Packt Publishing
ISBN: 9781788837996

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Dive deeper into neural networks and get your models trained, optimized with this quick reference guide

About This Book

  • A quick reference to all important deep learning concepts and their implementations
  • Essential tips, tricks, and hacks to train a variety of deep learning models such as CNNs, RNNs, LSTMs, and more
  • Supplemented with essential mathematics and theory, every chapter provides best practices and safe choices for training and fine-tuning your models in Keras and Tensorflow.

Who This Book Is For

If you are a Data Scientist or a Machine Learning expert, then this book is a very useful read in training your advanced machine learning and deep learning models. You can also refer this book if you are stuck in-between the neural network modeling and need immediate assistance in getting accomplishing the task smoothly. Some prior knowledge of Python and tight hold on the basics of machine learning is required.

What You Will Learn

  • Solve regression and classification challenges with TensorFlow and Keras
  • Learn to use Tensor Board for monitoring neural networks and its training
  • Optimize hyperparameters and safe choices/best practices
  • Build CNN's, RNN's, and LSTM's and using word embedding from scratch
  • Build and train seq2seq models for machine translation and chat applications.
  • Understanding Deep Q networks and how to use one to solve an autonomous agent problem.
  • Explore Deep Q Network and address autonomous agent challenges.

In Detail

Deep learning has become an essential necessity to enter the world of artificial intelligence. With this book deep learning techniques will become more accessible, practical, and relevant to practicing data scientists. It moves deep learning from academia to the real world through practical examples.

You will learn how Tensor Board is used to monitor the training of deep neural networks and solve binary classification problems using deep learning. Readers will then learn to optimize hyperparameters in their deep learning models. The book then takes the readers through the practical implementation of training CNN's, RNN's, and LSTM's with word embeddings and seq2seq models from scratch. Later the book explores advanced topics such as Deep Q Network to solve an autonomous agent problem and how to use two adversarial networks to generate artificial images that appear real. For implementation purposes, we look at popular Python-based deep learning frameworks such as Keras and Tensorflow, Each chapter provides best practices and safe choices to help readers make the right decision while training deep neural networks.

By the end of this book, you will be able to solve real-world problems quickly with deep neural networks.

Style and approach

An easy-to-follow, step-by-step guide to help you get to grips with real-world applications of training deep neural networks.

Table of contents

  1. Title Page
  2. Copyright and Credits
    1. Deep Learning Quick Reference
  3. Dedication
  4. Packt Upsell
    1. Why subscribe?
    2. PacktPub.com
  5. Foreword
  6. Contributors
    1. About the author
    2. About the reviewer
    3. Packt is searching for authors like you
  7. Preface
    1. Who this book is for
    2. What this book covers
    3. To get the most out of this book
      1. Download the example code files
      2. Conventions used
    4. Get in touch
      1. Reviews
  8. The Building Blocks of Deep Learning
    1. The deep neural network architectures
      1. Neurons
        1. The neuron linear function
        2. Neuron activation functions
      2. The loss and cost functions in deep learning
      3. The forward propagation process
      4. The back propagation function
      5. Stochastic and minibatch gradient descents
    2. Optimization algorithms for deep learning
      1. Using momentum with gradient descent
      2. The RMSProp algorithm
      3. The Adam optimizer
    3. Deep learning frameworks
      1. What is TensorFlow?
      2. What is Keras?
      3. Popular alternatives to TensorFlow
      4. GPU requirements for TensorFlow and Keras
      5. Installing Nvidia CUDA Toolkit and cuDNN
      6. Installing Python
      7. Installing TensorFlow and Keras
    4. Building datasets for deep learning
      1. Bias and variance errors in deep learning
      2. The train, val, and test datasets
      3. Managing bias and variance in deep neural networks
      4. K-Fold cross-validation
    5. Summary
  9. Using Deep Learning to Solve Regression Problems
    1. Regression analysis and deep neural networks
      1. Benefits of using a neural network for regression
      2. Drawbacks to consider when using a neural network for regression
    2. Using deep neural networks for regression
      1. How to plan a machine learning problem
      2. Defining our example problem
      3. Loading the dataset
      4. Defining our cost function
    3. Building an MLP in Keras
      1. Input layer shape
      2. Hidden layer shape
      3. Output layer shape
      4. Neural network architecture
      5. Training the Keras model
      6. Measuring the performance of our model
    4. Building a deep neural network in Keras
      1. Measuring the deep neural network performance
      2. Tuning the model hyperparameters
    5. Saving and loading a trained Keras model
    6. Summary
  10. Monitoring Network Training Using TensorBoard
    1. A brief overview of TensorBoard
    2. Setting up TensorBoard
      1. Installing TensorBoard
      2. How TensorBoard talks to Keras/TensorFlow
      3. Running TensorBoard
    3. Connecting Keras to TensorBoard
      1. Introducing Keras callbacks
      2. Creating a TensorBoard callback
    4. Using TensorBoard
      1. Visualizing training
      2. Visualizing network graphs
      3. Visualizing a broken network
    5. Summary
  11. Using Deep Learning to Solve Binary Classification Problems
    1. Binary classification and deep neural networks
      1. Benefits of deep neural networks
      2. Drawbacks of deep neural networks
    2. Case study – epileptic seizure recognition
      1. Defining our dataset
      2. Loading data
      3. Model inputs and outputs
      4. The cost function
      5. Using metrics to assess the performance
    3. Building a binary classifier in Keras
      1. The input layer
      2. The hidden layers
        1. What happens if we use too many neurons?
        2. What happens if we use too few neurons?
        3. Choosing a hidden layer architecture
        4. Coding the hidden layers for our example
      3. The output layer
      4. Putting it all together
      5. Training our model
    4. Using the checkpoint callback in Keras
    5. Measuring ROC AUC in a custom callback
    6. Measuring precision, recall, and f1-score
    7. Summary
  12. Using Keras to Solve Multiclass Classification Problems
    1. Multiclass classification and deep neural networks
      1. Benefits
      2. Drawbacks
    2. Case study - handwritten digit classification
      1. Problem definition
      2. Model inputs and outputs
        1. Flattening inputs
        2. Categorical outputs
      3. Cost function
      4. Metrics
    3. Building a multiclass classifier in Keras
      1. Loading MNIST
      2. Input layer
      3. Hidden layers
      4. Output layer
        1. Softmax activation
      5. Putting it all together
      6. Training
      7. Using scikit-learn metrics with multiclass models
    4. Controlling variance with dropout
    5. Controlling variance with regularization
    6. Summary
  13. Hyperparameter Optimization
    1. Should network architecture be considered a hyperparameter?
      1. Finding a giant and then standing on his shoulders
      2. Adding until you overfit, then regularizing
      3. Practical advice
    2. Which hyperparameters should we optimize?
    3. Hyperparameter optimization strategies
      1. Common strategies
      2. Using random search with scikit-learn
      3. Hyperband
    4. Summary
  14. Training a CNN from Scratch
    1. Introducing convolutions
      1. How do convolutional layers work?
        1. Convolutions in three dimensions
        2. A layer of convolutions
      2. Benefits of convolutional layers
        1. Parameter sharing
        2. Local connectivity
      3. Pooling layers
      4. Batch normalization
    2. Training a convolutional neural network in Keras
      1. Input
      2. Output
      3. Cost function and metrics
      4. Convolutional layers
      5. Fully connected layers
      6. Multi-GPU models in Keras
      7. Training
    3. Using data augmentation
      1. The Keras ImageDataGenerator
      2. Training with a generator
    4. Summary
  15. Transfer Learning with Pretrained CNNs
    1. Overview of transfer learning
    2. When transfer learning should be used
      1. Limited data
      2. Common problem domains
    3. The impact of source/target volume and similarity
      1. More data is always beneficial
      2. Source/target domain similarity
    4. Transfer learning in Keras
      1. Target domain overview
      2. Source domain overview
      3. Source network architecture
      4. Transfer network architecture
      5. Data preparation
      6. Data input
      7. Training (feature extraction)
      8. Training (fine-tuning)
    5. Summary
  16. Training an RNN from scratch
    1. Introducing recurrent neural networks
      1. What makes a neuron recurrent?
      2. Long Short Term Memory Networks
      3. Backpropagation through time
    2. A refresher on time series problems
      1. Stock and flow
      2. ARIMA and ARIMAX forecasting
    3. Using an LSTM for time series prediction
      1. Data preparation
        1. Loading the dataset
        2. Slicing train and test by date
        3. Differencing a time series
        4. Scaling a time series
        5. Creating a lagged training set
        6. Input shape
        7. Data preparation glue
      2. Network output
      3. Network architecture
      4. Stateful versus stateless LSTMs
      5. Training
      6. Measuring performance
    4. Summary
  17. Training LSTMs with Word Embeddings from Scratch
    1. An introduction to natural language processing
      1. Semantic analysis
      2. Document classification
    2. Vectorizing text
      1. NLP terminology
      2. Bag of Word models
      3. Stemming, lemmatization, and stopwords
      4. Count and TF-IDF vectorization
    3. Word embedding
      1. A quick example
      2. Learning word embeddings with prediction
      3. Learning word embeddings with counting
      4. Getting from words to documents
    4. Keras embedding layer
    5. 1D CNNs for natural language processing
    6. Case studies for document classifications
      1. Sentiment analysis with Keras embedding layers and LSTMs
        1. Preparing the data
        2. Input and embedding layer architecture
        3. LSTM layer
        4. Output layer
        5. Putting it all together
        6. Training the network
        7. Performance
      2. Document classification with and without GloVe
        1. Preparing the data
        2. Loading pretrained word vectors
        3. Input and embedding layer architecture
          1. Without GloVe vectors
          2. With GloVe vectors
        4. Convolution layers
        5. Output layer
        6. Putting it all together
        7. Training
        8. Performance
    7. Summary
  18. Training Seq2Seq Models
    1. Sequence-to-sequence models
      1. Sequence-to-sequence model applications
      2. Sequence-to-sequence model architecture
        1. Encoders and decoders
      3. Characters versus words
      4. Teacher forcing
      5. Attention
      6. Translation metrics
    2. Machine translation
      1. Understanding the data
      2. Loading data
      3. One hot encoding
      4. Training network architecture
      5. Network architecture (for inference)
      6. Putting it all together
      7. Training
      8. Inference
        1. Loading data
        2. Creating reverse indices
        3. Loading models
        4. Translating a sequence
        5. Decoding a sequence
        6. Example translations
    3. Summary
  19. Using Deep Reinforcement Learning
    1. Reinforcement learning overview
      1. Markov Decision Processes
      2. Q Learning
      3. Infinite state space
      4. Deep Q networks
        1. Online learning
        2. Memory and experience replay
      5. Exploitation versus exploration
      6. DeepMind
    2. The Keras reinforcement learning framework
      1. Installing Keras-RL
      2. Installing OpenAI gym
      3. Using OpenAI gym
    3. Building a reinforcement learning agent in Keras
      1. CartPole
        1. CartPole neural network architecture
        2. Memory
        3. Policy
        4. Agent
        5. Training
        6. Results
      2. Lunar Lander
        1. Lunar Lander network architecture
        2. Memory and policy
        3. Agent
        4. Training
        5. Results
    4. Summary
  20. Generative Adversarial Networks
    1. An overview of the GAN
    2. Deep Convolutional GAN architecture
      1. Adversarial training architecture
      2. Generator architecture
      3. Discriminator architecture
      4. Stacked training
        1. Step 1 – train the discriminator
        2. Step 2 – train the stack
    3. How GANs can fail
      1. Stability
      2. Mode collapse
    4. Safe choices for GAN
    5. Generating MNIST images using a Keras GAN
      1. Loading the dataset
      2. Building the generator
      3. Building the discriminator
      4. Building the stacked model
      5. The training loop
      6. Model evaluation
    6. Generating CIFAR-10 images using a Keras GAN
      1. Loading CIFAR-10
      2. Building the generator
      3. Building the discriminator
      4. The training loop
      5. Model evaluation
    7. Summary
  21. Other Books You May Enjoy
    1. Leave a review - let other readers know what you think

Product information

  • Title: Deep Learning Quick Reference
  • Author(s): Mike Bernico
  • Release date: March 2018
  • Publisher(s): Packt Publishing
  • ISBN: 9781788837996