json4s

NotSerializableException with json4s on Spark

与世无争的帅哥 提交于 2019-12-01 14:55:11
问题 Basically, i have to analyze some complex JSON's on HDFS with Spark. I use "for comprehensions" to (pre)filter the JSON's and "extract" method of json4s to wrap it into a case class This one works fine! def foo(rdd: RDD[String]) = { case class View(C: String,b: Option[Array[List[String]]], t: Time) case class Time($numberLong: String) implicit val formats = DefaultFormats rdd.map { jsonString => val jsonObj = parse(jsonString) val listsOfView = for { JObject(value) <- jsonObj JField(("v"),

How can I convert a json string to a scala map?

那年仲夏 提交于 2019-11-30 18:51:04
I have a nested json whose structure is not defined. It can be different each time I run since I am reading from a remote file. I need to convert this json into a map of type Map[String, Any] . I tried to look into json4s and jackson parsers but they don't seem to solve this issue I have. Does anyone know how I can achieve this? Example string: {"body":{ "method":"string", "events":"string", "clients":"string", "parameter":"string", "channel":"string", "metadata":{ "meta1":"string", "meta2":"string", "meta3":"string" } }, "timestamp":"string"} The level of nesting can be arbitrary and not

Combining type and field serializers

风格不统一 提交于 2019-11-30 17:44:05
Let's assume I have a case class with the following setup: case class Place(id:java.util.UUID, name:String) I can write a (working!) serializer for this type as follows: class placeSerializer extends CustomSerializer[Place]( format => ( { case JObject(JField("id", JString(s)) :: JField("name",JString(x)) :: Nil ) => Place(UUID.fromString(s), x) }, { case x:Place => JObject( JField("id", JString(x.id.toString())) :: JField("name", JString(x.name)) :: Nil) } ) ) But assuming my case class eventually has a lot more fields, this could lead to me enumerating the entire structure of the object with

How can I convert a json string to a scala map?

为君一笑 提交于 2019-11-30 02:32:40
问题 I have a nested json whose structure is not defined. It can be different each time I run since I am reading from a remote file. I need to convert this json into a map of type Map[String, Any] . I tried to look into json4s and jackson parsers but they don't seem to solve this issue I have. Does anyone know how I can achieve this? Example string: {"body":{ "method":"string", "events":"string", "clients":"string", "parameter":"string", "channel":"string", "metadata":{ "meta1":"string", "meta2":

Spark non-serializable exception when parsing JSON with json4s

亡梦爱人 提交于 2019-11-30 02:25:34
问题 I've run into an issue with attempting to parse json in my spark job. I'm using spark 1.1.0 , json4s , and the Cassandra Spark Connector . The exception thrown is: java.io.NotSerializableException: org.json4s.DefaultFormats Examining the DefaultFormats companion object, and with this stack question, it is clear that DefaultFormats cannot be serialized. The question is now what to do. I can see this ticket has apparently addressed this issue in the spark code base, by adding the keyword

Combining type and field serializers

冷暖自知 提交于 2019-11-30 01:46:56
问题 Let's assume I have a case class with the following setup: case class Place(id:java.util.UUID, name:String) I can write a (working!) serializer for this type as follows: class placeSerializer extends CustomSerializer[Place]( format => ( { case JObject(JField("id", JString(s)) :: JField("name",JString(x)) :: Nil ) => Place(UUID.fromString(s), x) }, { case x:Place => JObject( JField("id", JString(x.id.toString())) :: JField("name", JString(x.name)) :: Nil) } ) ) But assuming my case class

Is it possible to make json4s not to throw exception when required field is missing?

你说的曾经没有我的故事 提交于 2019-11-30 01:19:23
问题 Is it possible to make json4s not to throw exception when required field is missing ? When I "extract" object from raw json string it throws exception like this one org.json4s.package$MappingException: No usable value for pager No usable value for rpp Did not find value which can be converted into byte at org.json4s.reflect.package$.fail(package.scala:98) at org.json4s.Extraction$ClassInstanceBuilder.org$json4s$Extraction$ClassInstanceBuilder$$buildCtorArg(Extraction.scala:388) at org.json4s

Json4s ignoring None fields during seriallization (instead of using 'null')

眉间皱痕 提交于 2019-11-29 11:39:01
I am having a generic json serialization method that uses json4s . Unfortunately, it is ignoring fields if the value is None . My goal is to have None fields be represented with a null value. I tried by adding custom serializer for None, but still it is not working. object test extends App { class NoneSerializer extends CustomSerializer[Option[_]](format => ( { case JNull => None }, { case None => JNull })) implicit val f = DefaultFormats + new NoneSerializer case class JsonTest(x: String, y: Option[String], z: Option[Int], a: Option[Double], b: Option[Boolean], c:Option[Date], d: Option[Any])

Deserialization of case object in Scala with JSON4S

时光毁灭记忆、已成空白 提交于 2019-11-29 10:40:16
I have some case classes defined like follows: sealed trait Breed case object Beagle extends Breed case object Mastiff extends Breed case object Yorkie extends Breed case class Dog(name: String, breed: Breed) I also have an endpoint defined with Scalatra: post("/dog") { val dog = parsedBody.extract[Dog] ... } I'd like this JSON object: { name: "Spike", breed: "Mastiff" } to deserialize to the appropriate instance of Dog . I'm struggling to figure out how to write a custom deserializer for Breed and register it with JSON4S. You need to write the serializer like below: Serializer : case object

How to serialize object to AST using json4s?

断了今生、忘了曾经 提交于 2019-11-29 02:41:05
问题 I am writing a Customer Serializer. In that Serializer I would like to somehow say: "and this thing you already know how to serialize". My current approach looks like that: import org.json4s.native.Serialization._ import org.json4s.JsonDSL.WithBigDecimal._ object WindowSerializer extends CustomSerializer[Window](format => ( [omitted], { case Window(frame, size) => ( "size" -> size ) ~ ( "frame" -> parse(write(frame)) ) })) That parse(write(frame)) things is both ugly and inefficient. How to