如何使用 spark scala 计算 S3 中的目录数？

Question

I have a problem to count the number of the folder in a S3 specific directory with Spark Scala.

Directory structure:

source/20220309/client_1/file_1.csv
source/20220309/client_2/file_1.csv

source/20220308/client_1/file_1.csv
source/20220308/client_2/file_1.csv

so i am looking for count 20220309 and 20220308. So the count will be 2. I do not want count for nested folder just top level folder count.

Answer 1

You can use the hadoop.fs lib:

import org.apache.hadoop.fs.{FileSystem, Path}
import org.apache.hadoop.conf.Configuration
import java.net.URI
    
    
val path = "s3a://bucket/prefix"
val fs = FileSystem.get(URI.create(path), new Configuration())
val dirs = fs.listStatus(new Path(path)).count(_.isDirectory)