TypeError：數據應該是LabeledPoint的RDD，但是得到了<type 'numpy.ndarray'>

Question

我收到錯誤：

TypeError: data should be an RDD of LabeledPoint, but got <type 'numpy.ndarray'>

當我執行時：

import sys
import numpy as np
from pyspark import SparkConf, SparkContext
from pyspark.mllib.classification import LogisticRegressionWithSGD


conf = (SparkConf().setMaster("local")
.setAppName("Logistic Regression")
.set("spark.executor.memory", "1g"))
sc = SparkContext(conf = conf) 


def mapper(line):
    feats = line.strip().split(",") 
    label = feats[len(feats) - 1]       # Last column is the label
    feats = feats[2: len(feats) - 1]    # remove id and type column
    feats.insert(0,label)
    features = [ float(feature) for feature in feats ] # need floats
    return np.array(features)

data = sc.textFile("test.csv")
parsedData = data.map(mapper)

# Train model
model = LogisticRegressionWithSGD.train(parsedData)

我在model = LogisticRegressionWithSGD.train(parsedData)行上收到錯誤。

parsedData應該是RDD。 我不確定為什么要得到這個。

Github鏈接到完整源代碼

Answer 1

parsedData應該是RDD。 我不確定為什么要得到這個。

問題不是parsedData不是RDD ，而是它存儲的內容。 由於消息說，你需要RDD[LabeledPoint]當你通過RDD[numpy.ndarray]

from pyspark.mllib.regression import LabeledPoint

def mapper(line):
    ...
    return LabeledPoint(label, features)