Ruby parsing json throws an error Uncaught exception: 399: unexpected token

Question

When I use JSON.parse(response['data']) , Ruby raises the error:

Uncaught exception: 399: unexpected token at ...

And I am not sure why. In casperjs, I echo the following back to Ruby:

this.echo('{"data": '+ JSON.stringify(resultFinal) + '}');

I use JSON.stringify to encode the string, which yields the following:

{"data": "[{'Docket Type': 'MOT - MOTION','Filing Date': '19-AUG-2015','Filing Party': 'PNC BANK, NATIONAL ASSOCIATION,','Docket Text': 'TO GRANT WRIT OF POSSESSION FOR FAILURE TO VACATE PREMISES F\\B PLT, PNC BANK'},{'Docket Type': 'AFF - AFFIDAVIT','Filing Date': '19-AUG-2015','Filing Party': '&nbsp;','Docket Text': 'PURSUANT TO SECTION 83.561, FLORIDA STATUTES  F\\B PLT, PNC'},{'Docket Type': '108FF - CAFF/REOPEN ($50.00)','Filing Date': '19-AUG-2015','Filing Party': '&nbsp;','Docket Text': '<i>none.</i>'},{'Docket Type': 'RO - REOPEN','Filing Date': '19-AUG-2015','Filing Party': '&nbsp;','Docket Text': '<i>none.</i>'},{'Docket Type': 'COS - CERTIFICATE OF SERVICE','Filing Date': '14-JUL-2015','Filing Party': 'NATIONAL CITY BANK,','Docket Text': 'OF ORDER ON OBJECTION TO FORECLOSURE SALE F/B PLT'},{'Docket Type': 'CRT - CERTIFICATE','Filing Date': '01-JUL-2015','Filing Party': '&nbsp;','Docket Text': 'OF DISBURSEMENTS'},{'Docket Type': 'COFT - CERTIFICATE OF TITLE','Filing Date': '01-JUL-2015','Filing Party': '&nbsp;','Docket Text': 'WAS SOLD TO PNC BANK NATIONAL ASSOCIATION'}]"}

Properly encoded as a JSON string, Ruby's JSON.parse should be able to parse it. So why does it fail?

Answer 1

Testing the JSON shows no problems:

require 'json'

str = <<EOT
{"data": "[{'Docket Type': 'MOT - MOTION','Filing Date': '19-AUG-2015','Filing Party': 'PNC BANK, NATIONAL ASSOCIATION,','Docket Text': 'TO GRANT WRIT OF POSSESSION FOR FAILURE TO VACATE PREMISES F\\B PLT, PNC BANK'},{'Docket Type': 'AFF - AFFIDAVIT','Filing Date': '19-AUG-2015','Filing Party': '&nbsp;','Docket Text': 'PURSUANT TO SECTION 83.561, FLORIDA STATUTES F\\B PLT, PNC'},{'Docket Type': '108FF - CAFF/REOPEN ($50.00)','Filing Date': '19-AUG-2015','Filing Party': '&nbsp;','Docket Text': '<i>none.</i>'},{'Docket Type': 'RO - REOPEN','Filing Date': '19-AUG-2015','Filing Party': '&nbsp;','Docket Text': '<i>none.</i>'},{'Docket Type': 'COS - CERTIFICATE OF SERVICE','Filing Date': '14-JUL-2015','Filing Party': 'NATIONAL CITY BANK,','Docket Text': 'OF ORDER ON OBJECTION TO FORECLOSURE SALE F/B PLT'},{'Docket Type': 'CRT - CERTIFICATE','Filing Date': '01-JUL-2015','Filing Party': '&nbsp;','Docket Text': 'OF DISBURSEMENTS'},{'Docket Type': 'COFT - CERTIFICATE OF TITLE','Filing Date': '01-JUL-2015','Filing Party': '&nbsp;','Docket Text': 'WAS SOLD TO PNC BANK NATIONAL ASSOCIATION'}]"}
EOT

JSON[str]
# => {"data"=>
#      "[{'Docket Type': 'MOT - MOTION','Filing Date': '19-AUG-2015','Filing Party': 'PNC BANK, NATIONAL ASSOCIATION,','Docket Text': 'TO GRANT WRIT OF POSSESSION FOR FAILURE TO VACATE PREMISES FB PLT, PNC BANK'},{'Docket Type': 'AFF - AFFIDAVIT','Filing Date': '19-AUG-2015','Filing Party': '&nbsp;','Docket Text': 'PURSUANT TO SECTION 83.561, FLORIDA STATUTES FB PLT, PNC'},{'Docket Type': '108FF - CAFF/REOPEN ($50.00)','Filing Date': '19-AUG-2015','Filing Party': '&nbsp;','Docket Text': '<i>none.</i>'},{'Docket Type': 'RO - REOPEN','Filing Date': '19-AUG-2015','Filing Party': '&nbsp;','Docket Text': '<i>none.</i>'},{'Docket Type': 'COS - CERTIFICATE OF SERVICE','Filing Date': '14-JUL-2015','Filing Party': 'NATIONAL CITY BANK,','Docket Text': 'OF ORDER ON OBJECTION TO FORECLOSURE SALE F/B PLT'},{'Docket Type': 'CRT - CERTIFICATE','Filing Date': '01-JUL-2015','Filing Party': '&nbsp;','Docket Text': 'OF DISBURSEMENTS'},{'Docket Type': 'COFT - CERTIFICATE OF TITLE','Filing Date': '01-JUL-2015','Filing Party': '&nbsp;','Docket Text': 'WAS SOLD TO PNC BANK NATIONAL ASSOCIATION'}]"}

I'm on Ruby 2.2.3.

---

The problem is that the JSON string is being used to transfer another JSON string, which has had its expected double-quotes translated to single-quotes. Convert them back:

hash = JSON[str]
JSON[hash['data'].tr("'", '"')]
# => [{"Docket Type"=>"MOT - MOTION",
#      "Filing Date"=>"19-AUG-2015",
#      "Filing Party"=>"PNC BANK, NATIONAL ASSOCIATION,",
#      "Docket Text"=>
#       "TO GRANT WRIT OF POSSESSION FOR FAILURE TO VACATE PREMISES FB PLT, PNC BANK"},
#     {"Docket Type"=>"AFF - AFFIDAVIT",
#      "Filing Date"=>"19-AUG-2015",
#      "Filing Party"=>" ",
#      "Docket Text"=>
#       "PURSUANT TO SECTION 83.561, FLORIDA STATUTES FB PLT, PNC"},
#     {"Docket Type"=>"108FF - CAFF/REOPEN ($50.00)",
#      "Filing Date"=>"19-AUG-2015",
#      "Filing Party"=>" ",
#      "Docket Text"=>"<i>none.</i>"},
#     {"Docket Type"=>"RO - REOPEN",
#      "Filing Date"=>"19-AUG-2015",
#      "Filing Party"=>"&nbsp;",
#      "Docket Text"=>"<i>none.</i>"},
#     {"Docket Type"=>"COS - CERTIFICATE OF SERVICE",
#      "Filing Date"=>"14-JUL-2015",
#      "Filing Party"=>"NATIONAL CITY BANK,",
#      "Docket Text"=>"OF ORDER ON OBJECTION TO FORECLOSURE SALE F/B PLT"},
#     {"Docket Type"=>"CRT - CERTIFICATE",
#      "Filing Date"=>"01-JUL-2015",
#      "Filing Party"=>"&nbsp;",
#      "Docket Text"=>"OF DISBURSEMENTS"},
#     {"Docket Type"=>"COFT - CERTIFICATE OF TITLE",
#      "Filing Date"=>"01-JUL-2015",
#      "Filing Party"=>"&nbsp;",
#      "Docket Text"=>"WAS SOLD TO PNC BANK NATIONAL ASSOCIATION"}]

JSON is a defined format and must follow the specification . Strings are delimited with double-quotes:

A value can be a string in double quotes, or a number, or true or false or null, or an object or an array. These structures can be nested.

Encoding JSON in JSON is screwy but obviously someone is doing it (wrong). I think it should be base64 encoded to avoid the possibility of corruption or the need for translating the quotes, which is bound to fail or result in further machinations to get at the original data.

The alternate is to let the JSON parser serialize the data itself, which it will do using escaped characters:

require 'json'

foo = JSON[{'a' => 'b'}]
JSON[ {'bar' => foo }] # => "{\"bar\":\"{\\\"a\\\":\\\"b\\\"}\"}"

Which will look like:

{"bar":"{\"a\":\"b\"}"}

in real life.

Retrieving the data then becomes much more simple:

JSON[bar] # => {"bar"=>"{\"a\":\"b\"}"}
JSON[bar]['bar'] # => "{\"a\":\"b\"}"
JSON[bar]['bar']['a'] # => "a"

Having to decode multiple times is the "machinations" I refer to. And, the obvious downside to this is there'll be a growing "leaning-toothpick" situation happening, which can get pretty ugly but if nobody messes with them during the transfer of the data then JSON should be able to sort it out if enough decoding steps are done.