Pyspark：如何将在线.gz日志文件加载到pyspark.sql.dataframe.DataFrame中

Question

So I have a .gz log file hosted online, like this

https://example.com/sample.log.gz

I can load this into a Python list using this:

import urllib2
from StringIO import StringIO
import gzip

request = urllib2.Request('https://example.com/sample.log.gz')
response = urllib2.urlopen(request)
buf = StringIO(response.read())
f = gzip.GzipFile(fileobj=buf)
data = f.readlines() # Python list

I then tried to convert this list to DataFrame using

sqlContext.createDataFrame(data)

but got

TypeError: Can not infer schema for type: <type 'str'>

What would be an effective way to load the .gz log file directly into pyspark.sql.dataframe.DataFrame then?

Appreciate your help!

Answer 1

The problem comes from the form of your data variable. It is ['qwr', 'asd', 'wer'] but needs to be [['qwr'], ['asd'], ['wer']] .

To do so you can use data = [[x] for x in data]

Then sqlContext.createDataFrame(data)

---

Another solution could be to directly load your file as a textFile (however it requires to save the file somehow) then convert as presented above:

f = tempfile.NamedTemporaryFile(delete=True)
shutils.copyfileobj(response, f)
rdd = sc.textFile(f.name)
//save transformation as previously
rdd_list = rdd.map(lambda x: [x])
df = sqlContext.createDataFrame(rdd_list)