使用JSON模式更新Spark DataFrame中的列

Question

I have json files, and I'm trying to hash one field of it with SHA 256. These files are on AWS S3. I am currently using spark with python on Apache Zeppelin.

Here is my json schema, I am trying to hash 'mac' field;

 |-- Document: struct (nullable = true)
 |    |-- data: array (nullable = true)
 |    |    |-- element: struct (containsNull = true)
 |    |    |    |-- mac: string (nullable = true)

I've tried couple of things;

from pyspark.sql.functions import UserDefinedFunction
from pyspark.sql.types import StringType
import hashlib  

hcData = sqlc.read.option("inferSchema","true").json(inputPath)
hcData.registerTempTable("hcData")


name = 'Document'
udf = UserDefinedFunction(lambda x: hashlib.sha256(str(x).encode('utf-8')).hexdigest(), StringType())
new_df = hcData.select(*[udf(column).alias(name) if column == name else column for column in hcData.columns])

This code works fine. But when I try to hash mac field and change name variable nothing happens;

name = 'Document.data[0].mac'
name = 'mac'

I guess it is because, it couldn't find column with given name.

I've tried to change the code a bit;

def valueToCategory(value):
    return hashlib.sha256(str(value).encode('utf-8')).hexdigest()


udfValueToCategory = udf(valueToCategory, StringType())
df = hcData.withColumn("Document.data[0].mac",udfValueToCategory("Document.data.mac"))

This code hashes "Document.data.mac" and creates new column with hashed mac addresses. I want to update existing column. For those variables not nested it can update, there is no problem, but for nested variables I couldn't find a way to update.

So basically, I want to hash a field in nested json file with spark python. Can anyone knows how to update spark dataframe with schema ?

Answer 1

Well, I've found a solution for my question with scala . There can be redundant codes but it worked anyway.

import scala.util.matching.Regex
import java.security.MessageDigest

val inputPath = ""
val outputPath = ""

//finds mac addresses with given regex
def find(s: String, r: Regex): List[String] = {
    val l = r.findAllIn(s).toList
    if(!l.isEmpty){ 
        return l
    } else {
        val lis: List[String] = List("null")
        return lis
    }
}

//hashes given string with sha256
def hash(s: String): String = {
    return MessageDigest.getInstance("SHA-256").digest(s.getBytes).map(0xFF & _).map { "%02x".format(_) }.foldLeft(""){_ + _}
}

//hashes given line
def hashAll(s: String, r:Regex): String = {
    var st = s
    val macs = find(s, r)
    for (mac <- macs){
        st = st.replaceAll(mac, hash(mac))
    }
    return st
}

//read data
val rdd = sc.textFile(inputPath)

//mac address regular expression
val regex = "(([0-9A-Z]{1,2}[:-]){5}([0-9A-Z]{1,2}))".r

//hash data
val hashed_rdd = rdd.map(line => hashAll(line, regex))

//write hashed data
hashed_rdd.saveAsTextFile(outputPath)

Answer 2

Here is the python solution for my question below.

from pyspark.sql.functions import UserDefinedFunction
from pyspark.sql.types import StringType
import hashlib  
import re


def find(s, r):
    l = re.findall(r, s)
    if(len(l)!=0):
        return l
    else:
        lis = ["null"]
        return lis



def hash(s):
    return hashlib.sha256(str(s).encode('utf-8')).hexdigest()



def hashAll(s, r):
    st = s
    macs = re.findall(r, s)
    for mac in macs:
        st = st.replace(mac, hash(mac))
    return st


rdd = sc.textFile(inputPath)

regex = "([0-9A-Z]{1,2}[:-]){5}([0-9A-Z]{1,2})"
hashed_rdd = rdd.map(lambda line: hashAll(line, regex))

hashed_rdd.saveAsTextFile(outputPath)