Simple Hack To Keep SparkContext (and hence SparkUI) Alive After Job Completes

This example uses Scala syntax.

While programming in Spark on a local single node standalone cluster, you would be running your job either via spark-submit or running code in an IDE such as IntelliJ, NetBeans, Eclipse, etc.

 

After a job completes and your program exits, some activities that happen are:

– SparkContext is destroyed

Example log entry: 16/02/21 13:45:23 INFO SparkUI: Stopped Spark web UI at http://192.168.1.165:4040

– DAGScheduler is stopped

– MemoryStore is stopped, releasing your previously used memory

– BlockManager is stopped, hence all storage blocks used on disk for shuffle data, cached data spilled to disk, etc. are deleted

 

With the SparkContext, your SparkUI will also die. While coding on a local machine, if you do not run Spark on YARN or Mesos there would be no way to start the historyserver and retrieve historical jobs.

In such a case I use a simple hack to keep my SparkContext alive. I add the following command:

Thread.sleep(86400000); //sleep for 86400000 milliseconds, which is 24 hours

It may not be the smartest way but it does the job! My program will not exit for another 24 hours.

 

I've tried this method in the following cases:

1) When I want to observe in detail the steps happening in a specific action before my program proceeds. For example, if I add a sleep command after printing results of a countByValue() transformation:

val countOfWords=file.flatMap(word=>word.split("\n")).countByValue()

countOfWords.foreach(println);

Thread.sleep(86400000);

my job will print the output of countByValue() and then wait.

2) If I want to run a job and then go to sleep, but want to take a look at the SparkUI in the morning

 

Downside of this aproach – Because there is a sleep command in the code, your program will not exit until you kill it by force.