Extract Options from potentially null JSON values using for expression

Question

I have a JSON document where some values can be null. Using for expressions in json4s, how I can yield None, instead of nothing?

The following will fail to yield when the value for either of the fields FormattedID or PlanEstimate is null.

val j: json4s.JValue = ...
for {
  JObject(list) <- j
  JField("FormattedID", JString(id)) <- list
  JField("PlanEstimate", JDouble(points)) <- list
} yield (id, points)

For example:

import org.json4s._
import org.json4s.jackson.JsonMethods._

scala> parse("""{
     |   "FormattedID" : "the id",
     |   "PlanEstimate" : null
     | }""")
res1: org.json4s.JValue = JObject(List((FormattedID,JString(the id)), 
    (PlanEstimate,JNull)))

scala> for {                                      
     | JObject(thing) <- res1                     
     | JField("FormattedID", JString(id)) <- thing
     | } yield id                                 
res2: List[String] = List(the id)

scala> for {                                      
     | JObject(thing) <- res1                     
     | JField("PlanEstimate", JDouble(points)) <- thing
     | } yield points
res3: List[Double] = List()
// Ideally res3 should be List[Option[Double]] = List(None)

Answer 1

scala> object OptExtractors {
     |
     |   // Define a custom extractor for using in for-comprehension.
     |   // It returns Some[Option[Double]], instead of Option[Double].
     |   object JDoubleOpt {
     |     def unapply(e: Any) = e match {
     |       case d: JDouble => Some(JDouble.unapply(d))
     |       case _ => Some(None)
     |     }
     |   }
     | }
defined object OptExtractors

scala>

scala> val j = parse("""{
     |   "FormattedID" : "the id",
     |   "PlanEstimate" : null
     | }""")
j: org.json4s.JValue = JObject(List((FormattedID,JString(the id)), (PlanEstimate,JNull)))

scala>

scala> import OptExtractors._
import OptExtractors._

scala>

scala> for {
     |   JObject(list) <- j
     |   JField("FormattedID", JString(id)) <- list
     |   JField("PlanEstimate", JDoubleOpt(points)) <- list
     | } yield (id, points)
res1: List[(String, Option[Double])] = List((the id,None))

Answer 2

According to the documentation,

Any value can be optional. Field and value is completely removed when it doesn't have a value.

scala> val json = ("name" -> "joe") ~ ("age" -> (None: Option[Int]))

scala> compact(render(json))

res4: String = {"name":"joe"}

Explaining why your for comprehension doesn't yield anything.
Of course, a null value is mapped to None internally.

Answer 3

The last command should look like:

for {
  JObject(thing) <- res1
} yield thing.collectFirst{case JField("PlanEstimate", JDouble(points)) => points}

Or like

for {
  JObject(thing) <- res1
  points = thing.collectFirst{case JField("PlanEstimate", JDouble(p)) => p}
} yield points

Answer 4

What about this

 for {
      JObject(thing) <- res1      
      x = thing.find(_._1 == "PlanEstimate").flatMap(_._2.toOption.map(_.values))
     } yield x