如何使用Spark从文本文件中提取多行记录

Question

我有一个文本文件。 记录是所有多行记录，以“ \\ n \\ n \\ n \\ n \\ n”分隔。 文本文件如下所示：

name: Steven
gender: male
title: mr.
company: ABC

cell 647-777-****
home 905-000-****
work 289-***-1111





name: Al
gender: male
title: mr.
company: DEF

home 905-111-****
cell 289-991-****

我所做的是下面的代码：

val contact_raw = sc.wholeTextFiles("/user/data/contact.txt").flatMap(x => x._2.split("\n\n\n\n\n"))
val contact = contact_raw.map(contacts => {
    val per_person = contacts.split("\n\n")
    (per_person(0), per_person(1))
}).map(
    contact_info => {
        val personal_info = contact_info._1.split("\n")
        var name = ""
        var company = ""
        var gender = ""
        var title = ""
        for (x <- personal_info) {
            if(x.startsWith("name:")){
                name = x.split("name:")(1).trim
            } else if(x.startsWith("gender:")){
                gender = x.split("gender:")(1).trim
            } else if(x.startsWith("title:")){
                title = x.split("title:")(1)
            } else if(x.startsWith("company:")){
                company = x.split("company:")(1)
            } 
        }
        val phone_info = contact_info._2.split("\n").map(
                pair => {
                    val phone_pair = pair.split("\\s")
                    (phone_pair(0), phone_pair(1))
                }
            )
        (name, gender, title, company, phone_info)
    }
).toDF("name", "gender", "title", "company", "phone_info")

输出为：

scala> contact.show
+------+------+-----+-------+--------------------+
|  name|gender|title|company|          phone_info|
+------+------+-----+-------+--------------------+
|Steven|  male|  mr.|    ABC|[[cell,647-777-**...|
|    Al|  male|  mr.|    DEF|[[home,905-111-**...|
+------+------+-----+-------+--------------------+

架构是：

scala> contact.printSchema
root
 |-- name: string (nullable = true)
 |-- gender: string (nullable = true)
 |-- title: string (nullable = true)
 |-- company: string (nullable = true)
 |-- phone_info: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- _1: string (nullable = true)
 |    |    |-- _2: string (nullable = true)

预期的输出是：

+------+------+-----+-------+-------------+------------+
|  name|gender|title|company|   phone_type|number      |
+------+------+-----+-------+-------------+------------+
|Steven|  male|  mr.|    ABC|         cell|647-777-****|
|Steven|  male|  mr.|    ABC|         home|905-000-****|
|Steven|  male|  mr.|    ABC|         work|289-***-1111|
|    Al|  male|  mr.|    DEF|         home|905-111-****|
|    Al|  male|  mr.|    DEF|         cell|289-991-****|
+------+------+-----+-------+-------------+------------+

谁能告诉我如何修改代码以获得所需的输出？

Answer 1

以下将起作用：

val contact = contact_raw.map(contacts => {
      val per_person = contacts.split("\n\n")
      (per_person(0), per_person(1))
    }).flatMap(
      contact_info => {
        val personal_info = contact_info._1.split("\n")
        var name = ""
        var company = ""
        var gender = ""
        var title = ""
        for (x <- personal_info) {
          if (x.startsWith("name:")) {
            name = x.split("name:")(1).trim
          } else if (x.startsWith("gender:")) {
            gender = x.split("gender:")(1).trim
          } else if (x.startsWith("title:")) {
            title = x.split("title:")(1)
          } else if (x.startsWith("company:")) {
            company = x.split("company:")(1)
          }
        }
        contact_info._2.split("\n").map(
          pair => {
            val phone_pair = pair.split("\\s")
            (name, gender, title, company, phone_pair(0), phone_pair(1))
          }
        )
      }
    ).toDF("name", "gender", "title", "company", "phone_info")