SCIO read parquet file with scio parquet, the generated case class not found

Question

I have a problem.

I created a SCIO (Apache Beam) project via a sbt archetype: sbt new spotify/scio.g8

The goal of this Job is to Read a parquet file, from GS

When i use the ParquetIO provided by Apache Beam directly in SCIO, this work (GenericRecord):

val SCHEMA: Schema =
    new Schema.Parser()
      .parse(
        "{\n"
          + " \"namespace\": \"ioitavro\",\n"
          + " \"type\": \"record\",\n"
          + " \"name\": \"TestObject\",\n"
          + " \"fields\": [\n"
          + "     {\"name\": \"field1\", \"type\": \"int\"}\n,"
          + "     {\"name\": \"field2\", \"type\": \"int\"}\n"
          + " ]\n"
          + "}")

val inputIO = ParquetIO
 .read(SCHEMA)
 .from(options.getInputParquet)

sc.customInput("input", inputIO)
.map(file => file.toString)
.saveAsTextFile(args(outputTopic))

But i wanted to use the scio-parquet: https://spotify.github.io/scio/io/Parquet.html#read-avro-parquet-files

I have the following dependencies in the build.sbt file:

val scioVersion = "0.8.0-beta2"
val beamVersion = "2.16.0"
val scalaMacrosVersion = "2.1.1"

"com.spotify" %% "scio-core" % scioVersion,
"com.spotify" %% "scio-extra" % scioVersion,
"com.spotify" %% "scio-parquet" % scioVersion,
"com.spotify" %% "scio-avro" % scioVersion,

I use a schema

object TestModel {

  @AvroType.fromSchema(
    """{
      | "type":"record",
      | "name":"TestObject",
      | "namespace":"com.spotify.scio.avro",
      | "doc":"Record for an account",
      | "fields":[
      |   {"name":"field1","type":"int"},
      |   {"name":"field2","type":"int"}
      |  ]}
      """.stripMargin)
  class TestSchema

}

Then in the Job i use:

import com.spotify.scio.avro.TestObject

val projection = Projection[TestObject](_.getIntField, _.getIntField)

sc.parquetAvroFile[TestObject](args(inputParquet), projection)
  .map(file => file.toString)
  .saveAsTextFile(args(outputTopic))

But i have the following error:

object TestObject is not a member of package com.spotify.scio.avro
[error] import com.spotify.scio.avro.TestObject
[error]        ^
[error] /mypath/my-job/src/main/scala/com/test/MyJob.scala:41:33: not found: type TestObject
[error]     val projection = Projection[TestObject](_.getIntField, _.getIntField)
[error]                                 ^
[error] /my-path/my-job/src/main/scala/test/MyJob.scala:59:26: not found: type TestObject
[error]       sc.parquetAvroFile[TestObject](args(inputParquet), projection)
[error]                          ^

I don't understand why but maybe the case class TestObject is not generated correctly by the schema @AvroType.fromSchema.

Or maybe i don't use the api correctly, but i follow the link: https://spotify.github.io/scio/io/Parquet.html#read-avro-parquet-files

Thanks for your help.

Answer 1

With the sbt-avro plugin, this work.