Spark worker node

We are now ready to start our Spark Worker node, as follows:

> sbin/start-slave.sh spark://192.168.56.10:7077

Again, to check whether this was successful, you can examine the Spark logs as written to SPARK_LOG_DIR. You can also access the Spark Master web UI to confirm that the worker has been registered successfully, as illustrated in Figure 2.3:

Figure 2.3: Spark worker successfully registered

Note that Spark workers also expose a Worker UI via HTTP on port 8081 by default, in other words, http://<Worker IP Address>:8081.

Get Machine Learning with Apache Spark Quick Start Guide now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.