R-如何从JSON数据中删除特殊字符+转义序列，以便可以在其上使用JSONLITE？

Question

Going to just and provide a ton of info below with data, code, etc to finally get this solved. So first portion will just be an explanation.

Explanation - Program to automatically extract Tweets from Twitter. For reasons, I am storing certain values from them I care about (Description, Location, TweetID, etc) into a comma-delimited CSV file. I use HTTR to "GET" the tweets, the HTTR::content function to store them, then convert these to a more workable form using jsonlite::fromJSON(toJSON( )) . This works 90% of the time, but sometimes tweets have invisible escape characters embedded in them. Things like \\003 , GS as it appears in Notepad++, and other such special characters. These cause jsonlite to crash. I'd like to remove them.

So the code that works for 90% of the tweets looks like this:

mentions = GET(final_url, sig)
json = content(mentions)
json2 = jsonlite::fromJSON(toJSON(json))
allMentions = json2$statuses
colNames = names( unlist(allMentions[1,], use.names=TRUE ) )

Then a bunch more code to actually parse the tweets and pull out things like user_ids, text and latitude.

It crashes here at json2= line. With error:

Error: lexical error: invalid character inside string.
  Foundation and 42nd President of the United States. Follow 
              (right here) ------^

Or:

Error: lexical error: invalid character inside string.
   No Mission Too Difficult, No Sacrifice Too Great, Duty First. DAV, VFW.
                               (right here) ------^

So the first set of tweets that produces the first error, there is in notepad++ showing an escape character 003 after of. In the second, you can see a "GS" character after Great.

So the attempted fix was to use Gsub. Replace special characters. The problem was, then my data isn't in UTF-8 format anymore for some reason. So then I convert to UTF-8.

json = content(mentions)
json = gsub("[\001-\026]*", "", json, fixed=TRUE)
json = iconv(json, "UTF-8")
json2 = jsonlite::fromJSON(toJSON(json))
allMentions = json2$statuses

Now the jsonlite part works! Perfect, but not really.

Now I crash at "allMentions = json2$statuses"

    $ operator is invalid for atomic vectors

Which makes sense, because the output for json2 now is....

   [1] "NA"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
   [2] "list(completed_in = 0.131, max_id = 660500744261382144, max_id_str =        \"660500744261382146\", next_results = \"?      max_id=660499749334859776&q=%40HillaryClinton%20until%3A2015-11-     01&lang=en&count=100&include_entities=1&result_type=recent\", query = \"%40HillaryClinton+until%3A2015-11-01\", refresh_url = \"?since_id=660500744261382146&q=%40HillaryClinton%20until%3A2015-11-01&lang=en&result_type=recent&include_entities=1\", count = 100, since_id = 658634677922738176, since_id_str = \"658634677922738176\")"

This puts me at a loss. Do I persue fixing the error from this end now, and hunting down this, and risk this screwing up what was working for me previously? Do I go back to trying to figure out how to get rid of those Escape Characters / Invisible characters some other way?

Any advice on these errors would be met with much appreciation.

Answer 1

Ok, this might not be the perfect answer - I'm not experienced with R.

Regarding the bio of @pinoybreed808 :

No Mission Too Difficult, No Sacrifice Too Great, Duty First. DAV, VFW.

The unusual character appears after Duty First and before . it's actually U+001D - I've no idea why they're using it, but there are a couple of strategies for coping with unusual characters like this.

Firstly, you can simply gsub them out - although I'm not sure of R syntax, it doesn't look like you're capturing the character correctly.

Secondly, you could try URL Encoding the data before storing it. That would turn the character into %1D .

Thirdly, I don't see why this character is causing issues for JSONlite. It may be worth raising a bug in either R or JSONlite about how it copes with weird characters.