How to get Apache spark to ignore dots in a query?

Question

Given the following JSON file:

[{\"dog*woof\":\"bad dog 1\",\"dog.woof\":\"bad dog 32\"}]

Why does this Java code fail:

Dat         


        

Answer 1

It fails because dots are used to access attributes of the struct fields. You can escape column names using backticks:

val df = sqlContext.read.json(sc.parallelize(Seq(
   """{"dog*woof":"bad dog 1","dog.woof":"bad dog 32"}"""
)))

df.groupBy("`dog.woof`").count.show
// +----------+-----+
// |  dog.woof|count|
// +----------+-----+
// |bad dog 32|    1|
// +----------+-----+

but using special characters in the names is not a good practice and work with in general.