How to kill a Spark application

Spark applications can keep cluster cores and memory tied up after a mistaken submit, runaway query, or repeatedly failing supervised driver. Killing the correct driver submission stops that one application while leaving the standalone cluster and unrelated applications available.

In Spark standalone cluster mode, the kill target is the driver submission ID, not a job ID, stage ID, executor ID, or application UI ID. The submission ID looks like driver-20260706205619-0000 and is printed by spark-submit when the application is launched with --deploy-mode cluster.

Standalone REST submission management listens on port 6066 by default when the master is configured for it. The normal standalone master URL on port 7077 is still used for submitting work and connecting workers, but --kill and REST status requests must reach the submission endpoint.

1. Copy the driver submission ID from the cluster-mode submit output.

   INFO ClientEndpoint: Driver successfully submitted as driver-20260706205619-0000
   INFO ClientEndpoint: State of driver-20260706205619-0000 is RUNNING

   Use the driver-… value for standalone cancellation. The app-… value shown in the Spark UI identifies the application record but is not the value accepted by spark-submit --kill.

2. Check the driver state through the standalone REST submission endpoint.

   $ curl --silent http://spark-master.example.net:6066/v1/submissions/status/driver-20260706205619-0000
   {
     "action" : "SubmissionStatusResponse",
     "driverState" : "RUNNING",
     "serverSparkVersion" : "4.1.2",
     "submissionId" : "driver-20260706205619-0000",
     "success" : true,
     "workerHostPort" : "spark-worker.example.net:41933",
     "workerId" : "worker-20260706205600-spark-worker.example.net-41933"
   }

   If this request returns success: false or does not connect, check the submission ID and the standalone REST port before killing anything. A request sent to the 7077 RPC port is not a valid REST status request.

3. Kill the standalone driver submission with spark-submit.

   $ spark-submit --kill driver-20260706205619-0000 --master spark://spark-master.example.net:6066
   driver-20260706205619-0000 is killed successfully.

   This stops the whole Spark application for that driver. Confirm the submission ID before running the command on a shared cluster.

4. Verify that the driver state changed to KILLED.

   $ curl --silent http://spark-master.example.net:6066/v1/submissions/status/driver-20260706205619-0000
   {
     "action" : "SubmissionStatusResponse",
     "driverState" : "KILLED",
     "serverSparkVersion" : "4.1.2",
     "submissionId" : "driver-20260706205619-0000",
     "success" : true,
     "workerHostPort" : "spark-worker.example.net:41933",
     "workerId" : "worker-20260706205600-spark-worker.example.net-41933"
   }

5. Confirm that the application is no longer active on the standalone master.

   $ curl --silent http://spark-master.example.net:8080/json/
   {
     "url" : "spark://spark-master.example.net:7077",
   ##### snipped #####
     "activeapps" : [ ],
     "completedapps" : [ {
       "id" : "app-20260706205628-0000",
       "name" : "sg-kill-check",
       "state" : "FINISHED"
     } ],
     "activedrivers" : [ ],
     "completeddrivers" : [ {
       "id" : "driver-20260706205619-0000",
       "state" : "KILLED"
     } ],
     "status" : "ALIVE"
   }

   On YARN or Kubernetes, use the cluster manager's application-management path instead of the standalone master JSON. Spark on Kubernetes also accepts spark-submit --kill with a namespace:driver-pod-name submission ID.