如何在Apache Spark中删除以某些单词开头的多个hdfs目录

Question

I have persisted object files in spark streaming using dstream.saveAsObjectFiles("/temObj") method it shows multiple files in hdfs.

temObj-1506338844000
temObj-1506338848000
temObj-1506338852000
temObj-1506338856000
temObj-1506338860000

I want to delete all temObj files after reading all. What is the bet way to do it in spark. I tried

val hdfs = org.apache.hadoop.fs.FileSystem.get(new java.net.URI("hdfs://localhost:9000"), hadoopConf)
hdfs.delete(new org.apache.hadoop.fs.Path(Path), true) 

But it only can delete ane folder at a time

Answer 1

Unfortunately, delete doesn't support globs.

You can use globStatus and iterate over the files/directories one by one and delete them.

val hdfs = FileSystem.get(sc.hadoopConfiguration)

val deletePaths = hdfs.globStatus(new Path("/tempObj-*") ).map(_.getPath)

deletePaths.foreach{ path => hdfs.delete(path, true) }

Alternatively, you can use sys.process to execute shell commands

import scala.sys.process._

"hdfs dfs -rm -r /tempObj*" !