Spark - “sbt package” - “value $ is not a member of StringContext” - Missing Scala plugin?

Question

When running \"sbt package\" from the command line for a small Spark Scala application, I\'m getting the \"value $ is not a member of StringContext\" compilation error on the fo

Answer 1

In Spark 2.0+

$-notation for columns can be used by importing implicit on SparkSession object (spark)

val spark = org.apache.spark.sql.SparkSession.builder
        .master("local")
        .appName("App name")
        .getOrCreate;

import spark.implicits._

then your code with $ notation

val joined = ordered.join(empLogins, $"login" === $"username", "inner")
  .orderBy($"count".desc)
  .select("login", "count")