Moving Hadoop to the Cloud

Appendix A. Hadoop Component Start and Stop Scripts

Some Hadoop components, like HDFS and YARN, ship with management scripts that let you start and stop all of their component daemons or servers in one command. Others do not. The scripts here fill those gaps.

Tip

Code is available at this book’s code repository.

Apache ZooKeeper

The script in Example A-1 starts and stops Apache ZooKeeper servers, assuming each one is running on a worker instance in the cluster. It should be installed on the manager instance and run under the “zk” account, or any account that has passwordless SSH access to an account on each worker that may control ZooKeeper.

Example A-1. Control script for Apache ZooKeeper

#!/usr/bin/env bash

if [[ -z $1 ]]; then
  echo "Syntax: $0 (start|stop|status)"
  exit
fi

# Assumption is that each "slave" hosts a ZooKeeper server
SLAVES=( $(cat /etc/hadoop/slaves) )

case "$1" in
  start)
    echo Starting ZooKeeper
    for s in "${SLAVES[@]}"; do
      ssh "$s" /opt/zookeeper/bin/zkServer.sh start
    done
    ;;
  stop)
    echo Stopping ZooKeeper
    for s in "${SLAVES[@]}"; do
      ssh "$s" /opt/zookeeper/bin/zkServer.sh stop
    done
    ;;
  status)
    echo Checking ZooKeeper status
    for s in "${SLAVES[@]}"; do
      ssh "$s" /opt/zookeeper/bin/zkServer.sh status
    done
    ;;
esac

Apache Hive

The scripts in Examples A-2 and A-3 start and stop the Hive server (HiveServer2) and the Hive metastore server, respectively. They should be installed on an instance where Hive is installed, usually a manager, under an account such ...