将向量列表转换为PySpark中的DataFrame

Question

Firstly, I have load the data by:

import urllib.request
f = urllib.request.urlretrieve("https://www.dropbox.com/s/qz62t2oyllkl32s/kddcup.data_10_percent.gz?dl=1", "kddcup.data_10_percent.gz")


data_file = "./kddcup.data_10_percent.gz"
raw_data = sc.textFile(data_file)

Then, I created a list of required data by:

import numpy as np
import pandas as pd

def parse_interaction(line):
    line_split = line.split(",")
    # keep just numeric and logical values
    symbolic_indexes = [1,2,3,41]  # in the above sample would be: tcp,http,SF,normal
    clean_line_split = [item for i,item in enumerate(line_split) if i not in symbolic_indexes]
    return np.array([x for x in clean_line_split], dtype=float)

vector_data = raw_data.map(parse_interaction)

Now, I can see data by vector_data.take(2) :

[array([0.00e+00, 1.81e+02, 5.45e+03, 0.00e+00, 0.00e+00, 0.00e+00,
        0.00e+00, 0.00e+00, 1.00e+00, 0.00e+00, 0.00e+00, 0.00e+00,
        0.00e+00, 0.00e+00, 0.00e+00, 0.00e+00, 0.00e+00, 0.00e+00,
        0.00e+00, 8.00e+00, 8.00e+00, 0.00e+00, 0.00e+00, 0.00e+00,
        0.00e+00, 1.00e+00, 0.00e+00, 0.00e+00, 9.00e+00, 9.00e+00,
        1.00e+00, 0.00e+00, 1.10e-01, 0.00e+00, 0.00e+00, 0.00e+00,
        0.00e+00, 0.00e+00]),
 array([0.00e+00, 2.39e+02, 4.86e+02, 0.00e+00, 0.00e+00, 0.00e+00,
        0.00e+00, 0.00e+00, 1.00e+00, 0.00e+00, 0.00e+00, 0.00e+00,
        0.00e+00, 0.00e+00, 0.00e+00, 0.00e+00, 0.00e+00, 0.00e+00,
        0.00e+00, 8.00e+00, 8.00e+00, 0.00e+00, 0.00e+00, 0.00e+00,
        0.00e+00, 1.00e+00, 0.00e+00, 0.00e+00, 1.90e+01, 1.90e+01,
        1.00e+00, 0.00e+00, 5.00e-02, 0.00e+00, 0.00e+00, 0.00e+00,
        0.00e+00, 0.00e+00])]

I want to convert it into DataFrame with vector_data = pd.DataFrame(vector_data) , but the commands are not working and I am getting error, as:

ValueError           Traceback (most recent call last)
<ipython-input-112-6a2dcc5bdb85> in <module>()
     10 
     11 vector_data = raw_data.map(parse_interaction)
---> 12 vector_data = pd.DataFrame(vector_data)

~/anaconda3/lib/python3.6/site-packages/pandas/core/frame.py in __init__(self, data, index, columns, dtype, copy)
    420                                          dtype=values.dtype, copy=False)
    421             else:
--> 422                 raise ValueError('DataFrame constructor not properly called!')
    423 
    424         NDFrame.__init__(self, mgr, fastpath=True)

ValueError: DataFrame constructor not properly called!

I know that the input vector is in special format and I need to add something into DataFrame command to work properly. Please, guide me on that how can I make a DataFrame on that.

Answer 1

You can use from_records() :

vector_data = [np.array(...), np.array(...)]
pd.DataFrame.from_records(vector_data)