How to register custom Spark structured streaming source

Question

I need to create a custom streaming source by extending FileStreamSource. The idea is to override commit, so that processed files (S3 objects in this case) are renamed to have a certain prefix. However, I don't know how to use this custom source. Obviously I don't want to compile Spark -- the application will be running on Amazon EMR clusters.

Answer 1

Once you create your custom source in your project, you need to register it in the format of a DataStreamReader :

val input = spark
  .readStream
  .format("path.to.MyCustomFileStreamSource")
  .load