如何使用 spark 从 HTTP 源下载内容并将数据提取到 HDFS

Question

I have a use case to download content from an HTTP source and Ingest it to HDFS using python, the data available in the source is not live data, It has some content that changes once a week and I have to download the updated content every week, no of files to download would be 50k to 80k files and I have to do this by multithreading

I have a couple of questions

1. Can I use spark for this scenario? If so please tell me how to use it (or) point me to some spark resources

2. If spark is not a good fit for the scenario, then what else could I use?

Thanks in Advance

Answer 1

Spark would be a good choice if you want to move the file into HFDS.

This thread provides to way of achieving this: Adding a file to spark context:

 from pyspark import SparkFiles from pyspark.sql import SparkSession spark = SparkSession.builder.appName("test").getOrCreate() url = "https://dumps.wikimedia.org/other/clickstream/2017-11/clickstream-jawiki-2017-11.tsv.gz" spark.sparkContext.addFile(url) df = spark.read.option("sep", "\t").csv("file://" + SparkFiles.get("clickstream-jawiki-2017-11.tsv.gz")) df.show(10)

Using wget:

 import wget url = "https://dumps.wikimedia.org/other/clickstream/2017-11/clickstream-jawiki-2017-11.tsv.gz" local_path = '/tmp/wikipediadata/clickstream-jawiki-2017-11.tsv.gz' wget.download(url, local_path) df = spark.read.option("sep", "\t").csv('file://'+local_path) df.show(10)