I have a JSON file I want to read using Spark Scala, but when I read that file as DF it shows "_corrupt_record" column, and I tried all possible ways.
val df = spark.read
.format("json")
.option("multiline","true")
.load("PATH")
+--------------------+
| _corrupt_record|
+--------------------+
| {|
| "name": "Candace|
| "phone": "1-355...|
| "email": "egest...|
| "address": "160...|
| "postalZip": "1...|
| "rankings": "9,...|
| "alphanumeric":...|
| },|
| {|
| "name": "Grant ...|
| "phone": "(884)...|
| "email": "magna...|
| "address": "P.O...|
| "postal Zip": "6...|
| "rankings": "9,...|
| "alphanumeric":...|
| },|
| {|
| "name": "Patrice|
+--------------------+
Using the JSON example which you provided in the comments,
[ { "firstName": "Joe", "lastName": "Jackson", "gender": "male", "age": 28, "address": { "streetAddress": "101", "city": "San Diego", "state": "CA" }, "phoneNumbers": { "type": "home", "number": "7349282382" } } ]
the following line works well:
val df = spark.read.json("your/path")
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.