Spark streaming + json4s-jackson dependency problems

Question

I am unable to use json4s-Jackson 3.2.11 within my spark 1.4.1 Streaming application.

Thinking that it was the existing dependency within the spark-core project tha

Answer 1

Ok, I found the problem. As posted else where, you need to compile against jason4s 3.2.10. Apparently, doing so generates a binary which would then work with Spark (version 1.5 in my case. Same in some earlier versions as well). It has to do with the default parameter in the render() method which shows up in 3.2.11.

Answer 2

I had the same issue with emr 4.3.0 and spark 1.6 solved it with installing json4s in the bootstrap action by :

1. down load the json4s jar and put it in s3

2. create the following shell script and put it in s3

    #!/bin/bash
    set -e
    wget -S -T 10 -t 5 https://s3.amazonaws.com/your-bucketname/json4s-native_2.10-3.2.4.jar
    mkdir -p /home/hadoop/lib
    mv json4s-native_2.10-3.2.4.jar /home/hadoop/lib/   
   

3. add it as a bootstrap step in the emr launch steps