Scala/Spark App with “No TypeTag available” Error in “def main” style App

Question

I\'m new to Scala/Spark stack and I\'m trying to figure out how to test my basic skills using SparkSql to \"map\" RDDs in TempTables and viceversa.

I have 2 distinc

Answer 1

TL;DR;

Just move your case class out of the method definition

The problem is that your case class Log is defined inside of the method that it is being used. So, simply move your case class definition outside of the method and it will work. I will have to take a look at how this compiles down, but my guess is that this is more of a chicken-egg problem. The TypeTag (used for reflection) is not able to be implicitly defined as it has not been fully defined at that point. Here are two SO questions with the same problem that exhibit that Spark would need to use a WeakTypeTag. And, here is the JIRA explaining this more officially