Unexpected duration for v3io content delete

Question

When I deleted content of NoSqlTarget (key-value storage) in MLRun/v3io via standard command line utility such as:

rm -rf /v3io/project/test-featureset/

It took approx. 1 hour for 1 million of files and in case of 20 millions of files it was near to one day.

Do you have tips/tricks, how to use faster delete?

Answer 1

I got very useful tip for speedup delete process, use this command line:

hdfs dfs -rm -f -r v3io://projects/test-featureset/

The new performance was excellent, 1 million of files per 3 minutes.

BTW: the command hdfs dfs is possible to use for speedup other operations such as -ls -cp -find , more information see the link