特征错误 Spark Scala 中的重载方法

Question

I have some code

trait Reader {
  def read(spark: SparkSession, format: String, path: String): DataFrame
  def read[T: Encoder](spark: SparkSession, format: String, path: String): Dataset[T]
}

class LocalReader extends Reader {

  override def read[T: Encoder](spark: SparkSession, format: String, path: String): Dataset[T] = {
    spark.read
      .format(format)
      .option("header", "true")
      .load(getClass.getResource(path).getPath)
      .as[T]
  }

  override def read(spark: SparkSession, format: String, path: String): DataFrame = {
    spark.read
      .format(format)
      .option("header", "true")
      .load(getClass.getResource(path).getPath)
  }
}


object TopNSimilarCustomers extends SparkJob {
  override def appName: String = "TopNSimilarCustomers"

  override def run(spark: SparkSession, args: Array[String], reader: Reader): Unit = {

    /**
      * Only I/O here
      */

    if (args.length == 0)
      return
    val rawData = reader.read(spark, "json", "/spark-test-data.json")
    val res     = transform(spark, rawData, args(0))

  }

I'm getting an error at val rawData = reader.read(spark, "json", "/spark-test-data.json") cannot resolve overloaded method read.

So I want to have Readers/Writers for different purposes LocalReader/S3Reader and since it can return DF and DS I write an overloaded method even I have to use one. And eventually, have to implement both. Any way to avoid it?

How can I achieve what I'm trying to do? any other way or a better way etc? how to fix the error?

Answer 1

Reason for getting cannot resolve overloaded method read. is Reader trait has two methods both will take same number of params.

To solve this issue rename method names something like for example readDF & readDS or you can also check below code & modify as per your requirement.

    case class ReadConfig(format: String,path: String,options: Map[String,String])
    case class WriteConfig(format: String,path: String,options: Map[String,String])
    case class Config(read: ReadConfig,write: WriteConfig)

    trait Writer {
      def write(df: DataFrame): Unit
    }
    trait Reader {
      def read: DataFrame
    }

    trait RW extends Reader with Writer {
      val spark : SparkSession
      val config : Config
    }

    // Add logic for Local
    class Local(override val spark: SparkSession,override val config: Config) extends RW {
      override def read: DataFrame = {
        spark.read
          .format(config.read.format)
          .options(config.read.options)
          .load(config.read.path)
      }
      override def write(df: DataFrame): Unit = {
        df.write
          .format(config.write.format)
          .options(config.write.options)
          .save(config.write.path)
      }
    }

// Add logic for S3
 class S3(override val spark: SparkSession,override val config: Config) extends RW {
      override def read: DataFrame = {
        spark.read
          .format(config.read.format)
          .options(config.read.options)
          .load(config.read.path)
      }
      override def write(df: DataFrame): Unit = {
        df.write
          .format(config.write.format)
          .options(config.write.options)
          .save(config.write.path)
      }
    }