The web UI or Spark UI is a web interface for Spark components to monitor, analyze, and inspect job executions in a web browser. This allows for the monitoring of task progress and the timeline with the amount of resources given. Each and every component has its own web API, which has in-depth tracking of tasks in pending, processing, and completed statuses along with config, timeline, and logs. Make sure all of the required firewalls are opened to access the corresponding ports from all the instances in Spark cluster.
Port numbers and their usage:
- 4040: PySpark application port that is used to access in-depth details of an application that is running on a Spark cluster
- 4041: SparkR application port that is used to access in depth ...
