A Spark application's web UI can also supply the runtime measurements that monitoring systems collect. Spark's PrometheusServlet exposes driver metrics in Prometheus text format, allowing Prometheus to scrape job counts, memory use, and JVM measurements while the application runs.

The servlet is an experimental Spark metrics sink attached to the application's existing UI port. A custom metrics.properties file enables it for one application without changing the shared Spark installation, while a fixed metrics namespace keeps metric names stable across application restarts. Executor metrics exposed at a separate UI endpoint are outside this driver scrape.

A local PySpark shell provides a running application whose metrics remain available until the shell exits. Spark with Python and Java must already be installed, and an existing Prometheus server must be able to reach the driver over a private network. The Prometheus reload command requires its lifecycle API to be enabled on a protected listener because that API can also shut down the server.

Steps to enable Spark Prometheus metrics:

  1. Create metrics.properties in your current working directory using your text editor.
  2. Enable the Prometheus servlet and JVM source in metrics.properties.
    *.sink.prometheusServlet.class=org.apache.spark.metrics.sink.PrometheusServlet
    *.sink.prometheusServlet.path=/metrics/prometheus
    *.source.jvm.class=org.apache.spark.metrics.source.JvmSource

    The JVM source adds heap and garbage-collection measurements. Spark supplies scheduler and storage metrics without that optional source.

  3. Save metrics.properties.
  4. Start a local PySpark shell with the custom metrics file.
    $ "$SPARK_HOME/bin/pyspark" --master 'local[2]' \
      --conf spark.metrics.conf="$PWD/metrics.properties" \
      --conf spark.metrics.namespace=spark_metrics \
      --conf spark.ui.port=4040
    ##### snipped #####
    Spark context Web UI available at http://spark-driver:4040
    ##### snipped #####
    SparkSession available as 'spark'.
    >>>

    SPARK_HOME must point to your Spark installation. The UI hostname depends on your machine, and Spark tries the next port if 4040 is occupied. Keep this shell open during collection; an application's UI and metrics endpoint disappear when it stops. An existing application can use the same configuration options at its next submission.
    Related: How to submit an Apache Spark job

  5. Run a DataFrame action in the PySpark shell to generate job metrics.
    >>> spark.range(1000).count()
    1000
  6. Read the driver metrics from a second terminal on the Spark host.
    $ curl --fail --silent --show-error http://localhost:4040/metrics/prometheus/
    ##### snipped #####
    metrics_spark_metrics_driver_DAGScheduler_job_allJobs_Value{type="gauges"} 2
    ##### snipped #####

    A numeric job count under the spark_metrics namespace shows that the servlet is serving the running application. Counts and memory values vary with the workload. Use the UI port reported at startup if it differs from 4040. A Spark is starting up page is not metrics output.

  7. Open the active Prometheus configuration file in your text editor.
    $ sudoedit /etc/prometheus/prometheus.yml

    The active file is the one named by Prometheus's --config.file flag. The path shown is common for Linux services. Existing scrape jobs and other settings must remain intact.
    Related: How to add a Prometheus scrape config

  8. Add the Spark driver job under the existing scrape_configs list.
      - job_name: spark-driver
        metrics_path: /metrics/prometheus/
        static_configs:
          - targets: ['spark-driver:4040']

    The target must resolve to the driver host from Prometheus, with its actual UI port. Here spark-driver is the driver's private hostname; localhost would refer to the Prometheus host. Spark UI access can expose application details, so this change must not make the UI publicly accessible.

  9. Save the updated Prometheus configuration file.
  10. Validate the complete Prometheus configuration.
    $ promtool check config /etc/prometheus/prometheus.yml
    Checking /etc/prometheus/prometheus.yml
     SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

    The promtool binary ships with Prometheus. A failed check must be resolved before reload; referenced rule files may produce additional output.

  11. Reload Prometheus through its protected lifecycle endpoint.
    $ curl --fail --silent --show-error --request POST http://prometheus:9090/-/reload

    prometheus is the Prometheus server's hostname. A successful reload returns no body. A disabled lifecycle API requires the service manager's reload or SIGHUP method instead. To roll back, remove only the added spark-driver job and reload the configuration again.
    Related: How to reload Prometheus configuration

  12. Query the Spark job count stored in Prometheus after one scrape interval.
    $ promtool query instant http://prometheus:9090 \
      'metrics_spark_metrics_driver_DAGScheduler_job_allJobs_Value{job="spark-driver"}'
    metrics_spark_metrics_driver_DAGScheduler_job_allJobs_Value{instance="spark-driver:4040", job="spark-driver", type="gauges"} => 2 @[1794221573.264]

    The returned series proves that Prometheus collected a Spark driver metric, rather than merely accepting the configuration. The number and timestamp vary. An empty result means the metric has not been collected; up{job=“spark-driver”} equal to 0 indicates a failed scrape. Collection continues while the PySpark shell remains open.

Share this guide