简体   繁体   中英

How to read JSON file in Spark Scala?

I have a JSON file I want to read using Spark Scala, but when I read that file as DF it shows "_corrupt_record" column, and I tried all possible ways.

val df = spark.read
  .format("json")
  .option("multiline","true")
  .load("PATH")
+--------------------+
|     _corrupt_record|
+--------------------+
|                   {|
|    "name": "Candace|
|  "phone": "1-355...|
|  "email": "egest...|
|  "address": "160...|
|  "postalZip": "1...|
|  "rankings": "9,...|
|  "alphanumeric":...|
|                  },|
|                   {|
|  "name": "Grant ...|
|  "phone": "(884)...|
|  "email": "magna...|
|  "address": "P.O...|
| "postal Zip": "6...|
|  "rankings": "9,...|
|  "alphanumeric":...|
|                  },|
|                   {|
|    "name": "Patrice|
+--------------------+

Using the JSON example which you provided in the comments,

[ { "firstName": "Joe", "lastName": "Jackson", "gender": "male", "age": 28, "address": { "streetAddress": "101", "city": "San Diego", "state": "CA" }, "phoneNumbers": { "type": "home", "number": "7349282382" } } ]

the following line works well:

val df = spark.read.json("your/path")

在此处输入图像描述

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM