如何在 spark 中将 Avro Schema 对象转换为 StructType

Question

I have an RDD of type Row ie, RDD[Row] and avro schema object .I need to create a dataframe with this info.

I need toconvert avro schema object into StructType for creating DataFrame.

Can you please help .

Answer 1

com.databricks.spark.avro has a class to help you with this

 StructType requiredType = (StructType) SchemaConverters.toSqlType(AvroClass.getClassSchema()).dataType();

Please go through this specific example : http://bytepadding.com/big-data/spark/read-write-parquet-files-using-spark/

Answer 2

Updated as of 2020-05-31

Use below if you're on scala 2.12 with a newer spark version.

sbt:

scalaVersion := "2.12.11"
val sparkVersion = "2.4.5"
libraryDependencies += "org.apache.spark" %% "spark-avro" % sparkVersion

import org.apache.spark.sql.avro.SchemaConverters
import org.apache.spark.sql.types.StructType

val schemaType = SchemaConverters
  .toSqlType(avroSchema)
  .dataType
  .asInstanceOf[StructType]

Answer 3

Databrics gives support to avro related utilities in spark-avro package, use below dependency in sbt "com.databricks" % "spark-avro_2.11" % "3.2.0"

Code

*

val sqlSchema= SchemaConverters.toSqlType(avroSchema)

*

Before '3.2.0' version, 'toSqlType' is private method so if you are using older version than 3.2 then copy complete method in your own util class else upgrade to latest version.

Answer 4

Any example for doing same in pyspark? Below code works for me but there should be some other easier way to do this

# pyspark --packages org.apache.spark:spark-avro_2.11:2.4.4

import requests
import os
import avro.schema

from pyspark.sql.types import StructType

schema_registry_url = 'https://schema-registry.net/subjects/subject_name/versions/latest/schema'
schema_requests = requests.get(url=schema_registry_url)

spark_type = sc._jvm.org.apache.spark.sql.avro.SchemaConverters.toSqlType(sc._jvm.org.apache.avro.Schema.Parser().parse(schema_requests.text))

Answer 5

In pyspark 2.4.7 my solusion is to create an empty dataframe with avroschema and then take the the StructType object from this empty dataframe.

with open('/path/to/some.avsc','r') as avro_file:
    avro_scheme = avro_file.read()

df = spark\
    .read\
    .format("avro")\
    .option("avroSchema", avro_scheme)\
    .load()

struct_type = df.schema