Extracting data from list in R

Question

library(RCurl)
library(rjson)
json <- getURL('https://extraction.import.io/query/runtime/17d882b5-c118-4f27-8ce1-90085ec0b116?_apikey=d5a8a01e20174e95887dc0f385e4e3f6d7ef5ca1428d5a029f2aa352509948ade8e5d7fb0dc941f4769a32b541ca6b38a7cd6578dfd81b357fbc4f2e008f5154f1dbfcff31878798fa887b70b1ff59dd&url=http%3A%2F%2Fwww.numbeo.com%2Fcost-of-living%2Fcompare_cities.jsp%3Fcountry1%3DSingapore%26country2%3DAustralia%26city1%3DSingapore%26city2%3DMelbourne')
obj <- fromJSON(json)

I would like to get the data into nice columns of data, but many steps in the list are "nameless". Any idea of how to organise the data?

Answer 1

Check out this difference, and let me know what you think. This is what your object looks like:

library(RCurl)
library(rjson)
json <- getURL('https://extraction.import.io/query/runtime/17d882b5-c118-4f27-8ce1-90085ec0b116?_apikey=d5a8a01e20174e95887dc0f385e4e3f6d7ef5ca1428d5a029f2aa352509948ade8e5d7fb0dc941f4769a32b541ca6b38a7cd6578dfd81b357fbc4f2e008f5154f1dbfcff31878798fa887b70b1ff59dd&url=http%3A%2F%2Fwww.numbeo.com%2Fcost-of-living%2Fcompare_cities.jsp%3Fcountry1%3DSingapore%26country2%3DAustralia%26city1%3DSingapore%26city2%3DMelbourne')
obj <- rjson::fromJSON(json)
str(obj)

List of 2
 $ extractorData:List of 3
  ..$ url       : chr "http://www.numbeo.com/cost-of-living/compare_cities.jsp?country1=Singapore&country2=Australia&city1=Singapore&city2=Melbourne"
  ..$ resourceId: chr "b1250747011ee774e7c881617c86a5a9"
  ..$ data      :List of 1
  .. ..$ :List of 1
  .. .. ..$ group:List of 52
  .. .. .. ..$ :List of 6
  .. .. .. .. ..$ COL VALUE        :List of 1
  .. .. .. .. .. ..$ :List of 1
  .. .. .. .. .. .. ..$ text: chr "Meal, Inexpensive Restaurant"

Indeed a lot of Lists in between there that you don't need. Now try the jsonlite package's fromJSON function:

library(jsonlite)
obj2<- jsonlite::fromJSON(json)

List of 2
 $ extractorData:List of 3
  ..$ url       : chr "http://www.numbeo.com/cost-of-living/compare_cities.jsp?country1=Singapore&country2=Australia&city1=Singapore&city2=Melbourne"
  ..$ resourceId: chr "b1250747011ee774e7c881617c86a5a9"
  ..$ data      :'data.frame':  1 obs. of  1 variable:
  .. ..$ group:List of 1
  .. .. ..$ :'data.frame':  52 obs. of  6 variables:
  .. .. .. ..$ COL VALUE        :List of 52
  .. .. .. .. ..$ :'data.frame':    1 obs. of  1 variable:
  .. .. .. .. .. ..$ text: chr "Meal, Inexpensive Restaurant"
  .. .. .. .. ..$ :'data.frame':    1 obs. of  1 variable:
  .. .. .. .. .. ..$ text: chr "Meal for 2 People, Mid-range Restaurant, Three-course"
  .. .. .. .. ..$ :'data.frame':    1 obs. of  1 variable:

Still though, this JSON just isn't pretty, we'll need to fix this. I take it you want that data frame in there. So start with

df <- obj2$extractorData$data$group[[1]]

and there's your data frame. Problem though: every single cell is in a list here. Including NULL values, and you can't just unlist those, they'll disappear and the columns in which they were will grow shorter...

Edit: Here's how to handle the columns with list(NULL) values.

df[sapply(df[,2],is.null),2] <- NA
df[sapply(df[,3],is.null),3] <- NA
df[sapply(df[,4],is.null),4] <- NA
df[sapply(df[,5],is.null),5] <- NA
df2 <- sapply(df, unlist) %>% as.data.frame

It can be written more elegantly for sure, but this'll get you going and it's understandable.