Until now, we have worked with simple examples of running Spark workloads on Kubernetes from a local machine in this post. We also explained both spark-submit deployment modes when used from localhost in another post.

By now, you should already understand:

• what spark-submit does and how to use it
• how the driver lifecycle behaves based on the deploy mode

However, running Spark from your laptop is very different from running it in production. The key change is making the spark-submit process part of a Kubernetes workload.

We will achieve this by running it inside a Kubernetes Pod. By running spark-submit directly inside a Kubernetes Pod, Kubernetes can control it.

In client mode, the driver starts inside the same pod. In cluster mode, a separate driver pod is created.

Since both the submission process and the driver run as pods, their lifecycle is controlled by Kubernetes controllers. Understanding this ownership relationship is the key to predicting how Spark applications behave when pods are restarted, deleted, or redeployed.

The problem I was trying to solve when I first encountered this was handling long-running applications - mainly Spark Streaming. How do you redeploy them without leaving orphaned pods? What should you do with driver crashes or accidental deletions?

The example in this article is a simplified version of that scenario. We will intentionally terminate the submission process and observe how both deploy modes react. We will also delete the driver pod in cluster mode to examine how Kubernetes handles that lifecycle event.

Spark on Kubernetes Series

• ✅ Part 1: Introduction to Spark on Kubernetes
• ✅ Part 2: Spark Driver Lifecycle in Kubernetes
• 👉 Part 3: Spark on Kubernetes as a Managed Workload (current article)

With that in mind, let’s examine how this lifecycle coupling behaves in practice.

---

Long-running Applications Using Deployment

Long-running applications are those that do not have a finite goal. In Spark, these are typically streaming jobs that continuously read from a source, transform the data, and write it elsewhere.

The primary tool for working with long-running applications in Kubernetes is a Deployment.

A Kubernetes Deployment is a mechanism for managing pods and workloads. It ensures that the desired state is maintained. If you want one instance, there should always be one instance running. If your application fails, Kubernetes will restart it, as you will see later.

A Deployment manages a ReplicaSet, which allows you to scale applications up or down and manage pods automatically. You can stop, update, or restart an application without manually running spark-submit again.

To tighten our mental model, let’s describe how things look when we add a Deployment to the equation.

---

Client Mode Submit with Deployment inside Kubernetes

In client mode, the driver runs inside the deployment pod. spark-submit runs inside that pod. This pod is the driver, the spark-submit process owner, and the deployment instance at the same time.

Any changes to the Deployment will directly affect the Spark job.

This setup prevents orphaned drivers because deployment pods themselves cannot be orphaned. It also improves data consistency by avoiding multiple streams writing to the same data source.

---

Cluster Mode Submit with Deployment inside Kubernetes

In cluster mode, the driver runs in a separate pod outside the Deployment. spark-submit runs inside the deployment pod.

If the driver pod is killed while the deployment pod (running spark-submit) is still running, the deployment will resubmit the application and start a new driver.

If we stop the deployment pod (which runs spark-submit), the driver will continue running.

However, if the driver pod is deleted while the deployment pod is not running, it will not be automatically resubmitted. The driver has no controller watching it - only the spark-submit process is responsible for creating it.

From an ownership perspective, the deployment pod owns the spark-submit process. However, since the driver runs in a separate pod, stopping the deployment pod does not affect the driver.

This can feel confusing because multiple control layers are involved. Let’s separate them clearly.

---

Three Layers of Ownership

After this explanation, you might think that it’s actually not only about the driver lifecycle. And you would be right.

With the introduction of a Deployment, there are actually three different layers:

1. Kubernetes Deployment - manages pod replicas and owns the submit process
2. spark-submit process - submits applications, and may resubmit them; keeps the driver alive in cluster mode
3. Spark driver pod - manages executors

Before moving to the hands-on examples, let’s summarize the lifecycle:

Scenario	Cluster Mode	Client Mode
Kill driver	Resubmitted (if submit alive)	Whole app restarts
Kill submit	Driver keeps running	Whole app stops
Scale to 0	Driver survives	Everything stops

Now let’s verify these scenarios experimentally.

---

Hands-on Example of Deployment in Kubernetes

I chose NetworkWordCount from the Spark examples again. However, we need some additional setup:

$ minikube ssh # get into minikube container

# inside container run
$ nc -lk 9999 # netcat to listen on port 9999

This will generate a TCP stream that Spark Streaming can read continuously.

From this point on, whenever you see <your-minikube-ip>, replace it with your actual Minikube IP, since this is not executed from your shell and substitution will not work.

We will use the default namespace throughout. If you use a different namespace, adjust the configuration accordingly.

---

Deployment with Cluster Mode

This is the deployment we will use for testing. Save it as a YAML file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spark-test-deployment
  labels:
    app: spark
spec:
  replicas: 1
  selector:
    matchLabels:
      app: spark
  template:
    metadata:
      labels:
        app: spark
    spec:
      serviceAccountName: "spark"
      containers:
        - name: spark-test
          image: "apache/spark:4.0.1-scala2.13-java21-ubuntu"
          command: [ "sh", "-c" ]
          args:
            - |
              /opt/spark/bin/spark-submit \
                --master k8s://https://<your-minikube-ip>:8443 \
                --class org.apache.spark.examples.streaming.NetworkWordCount \
                --deploy-mode cluster \
                --name spark-network-word-count \
                --executor-memory 2G \
                --conf spark.executor.instances=2 \
                --conf spark.kubernetes.container.image=apache/spark:4.0.1-scala2.13-java21-ubuntu \
                --conf spark.kubernetes.authenticate.driver.serviceAccountName="spark" \
              local:///opt/spark/examples/jars/spark-examples.jar <your-minikube-ip> 9999

You don’t need to understand every detail here, but there are a few important points. First one is:

image: apache/spark:4.0.1-scala2.13-java21-ubuntu

This defines the Docker image. Kubernetes will use it to start pod within deployment. We use the same image for spark-submit, because it must run inside this pod.

command: [ "sh", "-c" ]
args: ...

This tells Kubernetes to start a shell and execute the provided arguments.

Changes made to the submit script:

• Updated --master to use the Minikube IP
• Changed the class to a streaming example
• Added required parameters after the JAR definition

Apply the deployment:

$ kubectl apply -f spark-deployment-test.yaml

Check the pods:

$ kubectl get pods -w
NAME                                       READY   STATUS    RESTARTS   AGE
networkwordcount-8390129c51527e66-exec-1   1/1     Running   0          28s
networkwordcount-8390129c51527e66-exec-2   1/1     Running   0          28s
spark-pi-7afec59c51527363-driver           1/1     Running   0          31s
spark-test-deployment-6dff6b9bb5-rnndp     1/1     Running   0          33s

We have:

• One deployment pod running spark-submit
• One driver coordinating the Spark cluster
• Two executor pods performing the work

---

Testing Driver Lifecycle in Cluster Mode

We will start by observing the driver lifecycle when we delete the driver pod.

Start watching pods in a separate shell:

$ kubectl get pods -w

And now delete the driver pod:

$ kubectl delete pods -l spark-role=driver

We are using the option -l spark-role=driver, which targets pods by label. This simplifies the command.

You should see something similar to this:

> k get pods -w
NAME                                               READY   STATUS              RESTARTS   AGE
networkwordcount-b163d39c80d5a5e5-exec-1           1/1     Running             0          52s
networkwordcount-b163d39c80d5a5e5-exec-2           1/1     Running             0          52s
spark-network-word-count-229bb19c80d58b0d-driver   1/1     Running             0          59s
spark-test-deployment-564b67b877-57kwv             1/1     Running             0          64s
spark-network-word-count-229bb19c80d58b0d-driver   1/1     Terminating         0          64s
networkwordcount-b163d39c80d5a5e5-exec-1           1/1     Terminating         0          58s
networkwordcount-b163d39c80d5a5e5-exec-2           1/1     Terminating         0          58s
...
spark-test-deployment-564b67b877-57kwv             0/1     Completed           0          71s
spark-test-deployment-564b67b877-57kwv             1/1     Running             1 (2s ago)   72s
spark-network-word-count-b05ca19c80d69d52-driver   0/1     Pending             0            0s
spark-network-word-count-b05ca19c80d69d52-driver   0/1     ContainerCreating   0            0s
spark-network-word-count-b05ca19c80d69d52-driver   1/1     Running             0            2s
networkwordcount-719f1e9c80d6b7b4-exec-1           0/1     Pending             0            0s
networkwordcount-719f1e9c80d6b7b4-exec-1           0/1     ContainerCreating   0            0s
networkwordcount-719f1e9c80d6b7b4-exec-2           0/1     ContainerCreating   0            0s
networkwordcount-719f1e9c80d6b7b4-exec-1           1/1     Running             0            1s
networkwordcount-719f1e9c80d6b7b4-exec-2           1/1     Running             0            2s

Lifecycle behavior:

1. The driver pod is terminated, which causes the termination of executor pods.
2. The deployment pod gets restarted, and the driver is resubmitted.
3. Executor pods start again.

That shows us that as long as the submit process in the deployment is running, the Spark job keeps running as well.

Notice that Kubernetes is not restarting the driver. It is restarting the spark-submit container, which then resubmits the application.

---

Now let’s see what happens when we scale the application down:

$ kubectl scale --replicas=0 deployments/spark-test-deployment

Check the pods:

$ kubectl get pods -n default
NAME                                       READY   STATUS    RESTARTS   AGE
networkwordcount-8390129c51527e66-exec-1   1/1     Running   0          55s
networkwordcount-8390129c51527e66-exec-2   1/1     Running   0          56s
spark-pi-7afec59c51527363-driver           1/1     Running   0          70s

The deployment pod disappears, meaning the spark-submit process is gone. The driver and executor pods are still running.

The driver is now a regular Kubernetes pod without a controller watching it, and killing the spark-submit process has no effect on it.

Let’s delete everything so that we have a clean state for the next steps:

$ kubectl delete -f spark-deployment-test.yaml
$ kubectl delete pods --all

---

Deployment with Client Mode

Let’s change the deploy mode to client.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spark-test-deployment
  labels:
    app: spark
spec:
  replicas: 1
  selector:
    matchLabels:
      app: spark
  template:
    metadata:
      labels:
        app: spark
    spec:
      serviceAccountName: "spark"
      containers:
        - name: spark-test
          image: "apache/spark:4.0.1-scala2.13-java21-ubuntu"
          command: [ "sh", "-c" ]
          args:
            - |
              /opt/spark/bin/spark-submit \
                --master k8s://https://<your-minikube-ip>:8443 \
                --class org.apache.spark.examples.streaming.NetworkWordCount \
                --deploy-mode client \
                --name spark-network-word-count \
                --executor-memory 2G \
                --conf spark.executor.instances=2 \
                --conf spark.kubernetes.container.image=apache/spark:4.0.1-scala2.13-java21-ubuntu \
                --conf spark.kubernetes.authenticate.driver.serviceAccountName="spark" \
              local:///opt/spark/examples/jars/spark-examples.jar <your-minikube-ip> 9999

Create this deployment:

$ kubectl apply -f spark-deployment-test.yaml

---

Executors will start throwing an error:

Caused by: java.io.IOException: Failed to connect to spark-test-deployment-9ff649856-27r5p/<unresolved>:40421   

In order for executors to work, they need to communicate with the driver. The driver coordinates work and distributes tasks between executors.

Right now, the executors cannot communicate back to the driver. Without that communication, they cannot run.

Delete previous deployment, because it will keep restarting:

$ kubectl delete -f spark-deployment-test.yaml

Let’s fix it. According to official documentation, we need to make configuration changes and add something called headless service.

---

Kubernetes Headless Service

In client mode, executors must be able to connect back to the driver pod.

Pods in Kubernetes have dynamic IP addresses. Executors need a stable way to connect to the driver pod.

A Kubernetes Service provides a stable network identity despite changing pod IPs.

In client mode we create a headless service (clusterIP: None). A headless service does not provide load balancing. Instead, it creates a stable DNS entry that resolves directly to the target pod.

To do that, save this code to some file called spark-service-test.yaml:

apiVersion: v1
kind: Service
metadata:
  name: spark
spec:
  clusterIP: None
  selector:
    app: spark
  ports:
    - protocol: TCP
      port: 23440
      targetPort: 23440

The app: spark label connects the Service and the Deployment. You could see the same label on Deployment as well.
The Service will route traffic to pods that match this label.

  ports:
    - protocol: TCP
      port: 23440
      targetPort: 23440

This configuration exposes port 23440 on the Service and forwards traffic to port 23440 on the driver pod.

Create it in Kubernetes:

$ kubectl create -f spark-service-test.yaml

Check if it was created:

$ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)     AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP     47h
spark        ClusterIP   None         <none>        23440/TCP   22h

We now have a Service named spark exposing port 23440. We will reference this Service in thespark-submit configuration later.

Changes for deployment

I will give you a final version of a Deployment, and then explain:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spark-test-deployment
  labels:
    app: spark
spec:
  replicas: 1
  selector:
    matchLabels:
      app: spark
  template:
    metadata:
      labels:
        app: spark
    spec:
      serviceAccountName: "spark"
      containers:
        - name: spark-test
          image: "apache/spark:4.0.1-scala2.13-java21-ubuntu"
          env:
            - name: SPARK_CONF_SPARK_KUBERNETES_DRIVER_POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: "metadata.name"
          command: [ "sh", "-c" ]
          args:
            - |
              /opt/spark/bin/spark-submit \
                --master k8s://https://<your-minikube-ip>:8443 \
                --class org.apache.spark.examples.streaming.NetworkWordCount \
                --deploy-mode client \
                --name spark-network-word-count \
                --executor-memory 2G \
                --conf spark.executor.instances=2 \
                --conf spark.kubernetes.container.image=apache/spark:4.0.1-scala2.13-java21-ubuntu \
                --conf spark.kubernetes.authenticate.driver.serviceAccountName="spark" \
                --conf spark.kubernetes.driver.pod.name=${SPARK_CONF_SPARK_KUBERNETES_DRIVER_POD_NAME} \
                --conf spark.driver.host=spark.default.svc \
                --conf spark.driver.port=23440 \
              local:///opt/spark/examples/jars/spark-examples.jar <your-minikube-ip> 9999

Env section

Pod names in Kubernetes are created at the time of their start. To capture them, we need to use special Kubernetes syntax.
This configuration takes that name and saves it to the environment variable named SPARK_CONF_SPARK_KUBERNETES_DRIVER_POD_NAME. We use it later in spark-submit.

Changes to spark submit

We added configuration:

--conf spark.kubernetes.driver.pod.name=${SPARK_CONF_SPARK_KUBERNETES_DRIVER_POD_NAME}

This is required configuration from documentation. We use env variable, we described in previous section. It helps Kubernetes coordinate cleaning of the pods after job is finished - garbage collection.

Remember the headless service we created? We tell spark how to find it by:

--conf spark.driver.host=spark.default.svc

Services in Kubernetes are defined by ..svc, it is internal kubernetes reference used instead of internal IPs as those are dynamic. The name of our headless service was spark, and we are using default namespace this whole time. Svc is service reference in kubernetes.

In headless service we also mentioned ports and those are used in following configuration:

--conf spark.driver.port=23440

It tells spark-submit which port to use with service defined in previous command.

Final Test and Results

Run the deployment:

$ kubectl apply -f spark-deployment-test.yaml

We will get:

$ kubectl get pods
NAME                                       READY   STATUS    RESTARTS   AGE
networkwordcount-9d863e9c5c787d93-exec-1   1/1     Running   0          23s
networkwordcount-9d863e9c5c787d93-exec-2   1/1     Running   0          22s
spark-test-deployment-698d8fccbb-mrlxg     1/1     Running   0          28s

This shows one deployment pod (driver) and two executor pods.

Now comes the key test: scale down the deployment.

$ kubectl scale --replicas=0 deployments/spark-test-deployment

Check the pods:

$ kubectl get pods
No resources found in default namespace.

All pods are terminated.

In client mode, the deployment pod owns the spark-submit process, so scaling it down stops the application entirely.

Kubernetes now manages the full lifecycle of your Spark job, just like any other service.

We can start it again by scaling the deployment up. Since the deployment contains the spark-submit command, it will start the job exactly as before.

$ kubectl scale --replicas=1 deployments/spark-test-deployment

$ kubectl get pods
NAME                                       READY   STATUS    RESTARTS   AGE
networkwordcount-53a63e9c5c787d93-exec-1   1/1     Running   0          15s
networkwordcount-53a63e9c5c787d93-exec-2   1/1     Running   0          14s
spark-test-deployment-554d8fccbb-mrlxg     1/1     Running   0          20s

---

Advantages of Client Deploy Mode for Long-running Applications

Advantages

• Direct control over application lifecycle
• Simpler runtime topology (no separate driver pod)
• Easier log collection and monitoring

Trade-offs

• More initial setup required
• A deeper understanding of Spark/Kubernetes interaction is necessary

---

Cleaning Up Pods

# delete deployment
$ kubectl delete -f spark-deployment-test.yaml

# delete service
$ kubectl delete -f spark-service-test.yaml

# delete all pods
$ kubectl delete pods --all

---

Wrapping Up: Driver Lifecycle and Deployments

Treating a Spark job as a Kubernetes workload means Kubernetes manages the lifecycle of the submission process - and indirectly the driver.

In return, we gain much more control over how different types of applications behave and are managed.

Client mode works well for long-running applications, particularly when you need scalability. It simplifies automation and redeployment. By binding the driver lifecycle to the deployment lifecycle, we gain predictable restart behavior and operational control.

It also prevents orphaned drivers or executors from being left behind.

In combination with production-level Delta Lake and properly configured streaming checkpoints, this approach also enables strong data consistency and safe restarts.

Cluster mode is simpler from a developer’s standpoint and makes deployment easier. However, it is less suited for long-running applications.

Kubernetes manages the driver lifecycle, but changes to the spark-submit process running in a separate pod do not affect the driver.

Finished driver pods may accumulate, which is sometimes desirable if you want a history of job runs.

This mode works well for batch jobs with a defined end, such as generating and sending daily reports.

For recurring jobs, consider using a Kubernetes CronJob.

There are many details and edge cases you’ll encounter as you explore further. Each use case may require a different approach, so experimenting hands-on is the best way to learn.

Once you understand who owns what, Spark on Kubernetes becomes predictable instead of mysterious.

Continue Reading

• ← Previous: Spark Driver Lifecycle in Kubernetes