Transposing JSON list-of-dictionaries for analysis in R

Question

I have experimental data expressed as dicts of key-value pairs for each experiment. A set of related experiments is serialized as a list of these dicts in JSON. This is parseabl

Answer 1

This is interesting. The easiest way would be to fix the Python code so that the dict can be transformed more easily.

But, how about this?

k1 <- unlist(lapply(data,FUN=function(x){return(x[[1]])}))
k2 <- unlist(lapply(data,FUN=function(x){return(x[[2]])}))
data.frame(k1,k2)

You will need to cast k1 and k2 into the correct data type still, but this should accomplish what you are looking for.

Answer 2

The l*ply functions can be your best friend when doing with list processing. Try this:

> library(plyr)
> ldply(data, data.frame)
  k1 k2
1 v1 v2
2 v3 v4

plyr does some very nice processing behind the scenes to deal with things like irregular lists (e.g. when each list doesn't contain the same number of elements). This is very common with JSON and XML, and is tricky to handle with the base functions.

Or alternatively using base functions:

> do.call("rbind", lapply(data, data.frame))

You can use rbind.fill (from plyr) instead of rbind if you have irregular lists, but I'd advise just using plyr from the beginning to make your life easier.

Edit:

Regarding your more complicated example, using Hadley's suggestion deals with this easily:

> x<-list(list(k1=2,k2=3),list(k2=100,k1=200),list(k1=5, k3=9))
> ldply(x, data.frame)
   k1  k2 k3
1   2   3 NA
2 200 100 NA
3   5  NA  9