Diffrence between extends from App and object contain main method in scala

Question

I wrote sample spark-scala program for creating list of json elements from dataframe. when i executed with main method it returns empty list but when i executed without obje

Answer 1

TL;DR Both snippets are not correct Spark programs, but one is just more incorrect than the other.

You've made two mistakes, both explained in the introductory Spark materials.

• Due to it's nature Spark doesn't support applications extending App - Quick Start - Self-Contained Applications

  Note that applications should define a main() method instead of extending scala.App. Subclasses of scala.App may not work correctly.

• Spark doesn't provide global shared memory therefore modifying global object is a closure is not supported - Spark Programming Guide - Understanding Closures