Spark：KMeans - ValueError：无法将字符串转换为浮点数：'0\x00\x00'
[英]Spark: KMeans - ValueError: could not convert string to float: '0\x00\x00'
问题描述
我正在尝试为 mnist 数据集创建一个 kmeans。 我有一种方法，但它是最肮脏的黑客。
我的输入是一个 CSV 文件，每行有 784 (=28*28) 个值，介于 0 和 255 之间。
我的第一次尝试是读取我的 csv 输入，将其转换为稀疏 arrays 并将 model 与数据相匹配。 但是，下面的代码会引发错误：
data = spark.read.csv("datasets/mnist_test.csv").rdd\
.map(lambda x : [float(str) for str in x])\
.toDF()
features = VectorAssembler(inputCols=data.columns, outputCol="features").transform(data).select("features")
kmeans = KMeans().setK(10).setSeed(1)
model = kmeans.fit(features)
Output：
22/01/25 10:44:41 ERROR Executor: Exception in task 4.0 in stage 113.0 (TID 131)
org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/opt/spark/python/lib/pyspark.zip/pyspark/worker.py", line 619, in main
process()
File "/opt/spark/python/lib/pyspark.zip/pyspark/worker.py", line 611, in process
serializer.dump_stream(out_iter, outfile)
File "/opt/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 259, in dump_stream
vs = list(itertools.islice(iterator, batch))
File "/opt/spark/python/lib/pyspark.zip/pyspark/util.py", line 74, in wrapper
return f(*args, **kwargs)
File "/tmp/ipykernel_74/2701217925.py", line 2, in <lambda>
File "/tmp/ipykernel_74/2701217925.py", line 2, in <listcomp>
ValueError: could not convert string to float: '0\x00\x00'
...
我的下一个尝试是将 dataframe 保存为 svm，然后再次加载：
MLUtils.saveAsLibSVMFile(features.rdd.map(lambda x: LabeledPoint(0, MLLibVectors.fromML(x.features))), './libsvm')
data2 = MLUtils.loadLibSVMFile(spark.sparkContext, './libsvm').toDF()
kmeans = KMeans().setK(10).setSeed(1)
model = kmeans.fit(features)
Output：
22/01/25 10:47:06 ERROR Instrumentation: java.lang.IllegalArgumentException: requirement failed: Column features must be of type equal to one of the following types: [struct<type:tinyint,size:int,indices:array<int>,values:array<double>>, array<double>, array<float>] but was actually of type struct<type:tinyint,size:int,indices:array<int>,values:array<double>>.
我的最后（工作）尝试是使用spark.read.format("libsvm").load(...)方法加载导出的分区：
data3 = spark.read.format("libsvm").load("libsvm/part-00000").select("features")
data3arr = list()
for i in range(5):
data3arr.append(spark.read.format("libsvm").load("libsvm/part-0000"+str(i)).select("features"))
data3cpl = data3arr[0]
for i in data3arr[1:]:
data3cpl.union(i)
kmeans = KMeans().setK(10).setSeed(1)
model = kmeans.fit(data3cpl)
如果我查看结构，数据框的结构看起来非常相似。 只有这些features在.show()上给我一个错误：
features.printSchema()
features.show(1,False)
data2.printSchema()
data2.show(1,False)
data3cpl.printSchema()
data3cpl.show(1,False)
Output：
root
|-- features: vector (nullable = true)
Traceback (most recent call last):
File "/opt/spark/python/lib/pyspark.zip/pyspark/daemon.py", line 186, in manager
File "/opt/spark/python/lib/pyspark.zip/pyspark/daemon.py", line 74, in worker
File "/opt/spark/python/lib/pyspark.zip/pyspark/worker.py", line 663, in main
if read_int(infile) == SpecialLengths.END_OF_STREAM:
File "/opt/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 564, in read_int
raise EOFError
EOFError
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|features |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|(784,[202,203,204,205,206,207,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,291,292,293,294,295,296,297,298,299,300,301,326,327,328,329,353,354,355,356,381,382,383,384,408,409,410,411,436,437,438,439,463,464,465,466,491,492,493,518,519,520,521,545,546,547,548,572,573,574,575,576,600,601,602,603,627,628,629,630,631,655,656,657,658,682,683,684,685,686,710,711,712,713,714,738,739,740,741],[84.0,185.0,159.0,151.0,60.0,36.0,222.0,254.0,254.0,254.0,254.0,241.0,198.0,198.0,198.0,198.0,198.0,198.0,198.0,198.0,170.0,52.0,67.0,114.0,72.0,114.0,163.0,227.0,254.0,225.0,254.0,254.0,254.0,250.0,229.0,254.0,254.0,140.0,17.0,66.0,14.0,67.0,67.0,67.0,59.0,21.0,236.0,254.0,106.0,83.0,253.0,209.0,18.0,22.0,233.0,255.0,83.0,129.0,254.0,238.0,44.0,59.0,249.0,254.0,62.0,133.0,254.0,187.0,5.0,9.0,205.0,248.0,58.0,126.0,254.0,182.0,75.0,251.0,240.0,57.0,19.0,221.0,254.0,166.0,3.0,203.0,254.0,219.0,35.0,38.0,254.0,254.0,77.0,31.0,224.0,254.0,115.0,1.0,133.0,254.0,254.0,52.0,61.0,242.0,254.0,254.0,52.0,121.0,254.0,254.0,219.0,40.0,121.0,254.0,207.0,18.0])|
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
only showing top 1 row
root
|-- features: vector (nullable = true)
|-- label: double (nullable = true)
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----+
|features |label|
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----+
|(778,[202,203,204,205,206,207,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,291,292,293,294,295,296,297,298,299,300,301,326,327,328,329,353,354,355,356,381,382,383,384,408,409,410,411,436,437,438,439,463,464,465,466,491,492,493,518,519,520,521,545,546,547,548,572,573,574,575,576,600,601,602,603,627,628,629,630,631,655,656,657,658,682,683,684,685,686,710,711,712,713,714,738,739,740,741],[84.0,185.0,159.0,151.0,60.0,36.0,222.0,254.0,254.0,254.0,254.0,241.0,198.0,198.0,198.0,198.0,198.0,198.0,198.0,198.0,170.0,52.0,67.0,114.0,72.0,114.0,163.0,227.0,254.0,225.0,254.0,254.0,254.0,250.0,229.0,254.0,254.0,140.0,17.0,66.0,14.0,67.0,67.0,67.0,59.0,21.0,236.0,254.0,106.0,83.0,253.0,209.0,18.0,22.0,233.0,255.0,83.0,129.0,254.0,238.0,44.0,59.0,249.0,254.0,62.0,133.0,254.0,187.0,5.0,9.0,205.0,248.0,58.0,126.0,254.0,182.0,75.0,251.0,240.0,57.0,19.0,221.0,254.0,166.0,3.0,203.0,254.0,219.0,35.0,38.0,254.0,254.0,77.0,31.0,224.0,254.0,115.0,1.0,133.0,254.0,254.0,52.0,61.0,242.0,254.0,254.0,52.0,121.0,254.0,254.0,219.0,40.0,121.0,254.0,207.0,18.0])|0.0 |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----+
only showing top 1 row
root
|-- features: vector (nullable = true)
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|features |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|(776,[202,203,204,205,206,207,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,291,292,293,294,295,296,297,298,299,300,301,326,327,328,329,353,354,355,356,381,382,383,384,408,409,410,411,436,437,438,439,463,464,465,466,491,492,493,518,519,520,521,545,546,547,548,572,573,574,575,576,600,601,602,603,627,628,629,630,631,655,656,657,658,682,683,684,685,686,710,711,712,713,714,738,739,740,741],[84.0,185.0,159.0,151.0,60.0,36.0,222.0,254.0,254.0,254.0,254.0,241.0,198.0,198.0,198.0,198.0,198.0,198.0,198.0,198.0,170.0,52.0,67.0,114.0,72.0,114.0,163.0,227.0,254.0,225.0,254.0,254.0,254.0,250.0,229.0,254.0,254.0,140.0,17.0,66.0,14.0,67.0,67.0,67.0,59.0,21.0,236.0,254.0,106.0,83.0,253.0,209.0,18.0,22.0,233.0,255.0,83.0,129.0,254.0,238.0,44.0,59.0,249.0,254.0,62.0,133.0,254.0,187.0,5.0,9.0,205.0,248.0,58.0,126.0,254.0,182.0,75.0,251.0,240.0,57.0,19.0,221.0,254.0,166.0,3.0,203.0,254.0,219.0,35.0,38.0,254.0,254.0,77.0,31.0,224.0,254.0,115.0,1.0,133.0,254.0,254.0,52.0,61.0,242.0,254.0,254.0,52.0,121.0,254.0,254.0,219.0,40.0,121.0,254.0,207.0,18.0])|
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
only showing top 1 row
谁能告诉我如何正确转换我的数据，以便我可以将其输入到我的 kmeans fit 中？
1 个解决方案
解决方案1
0 2022-08-03 17:32:23
我正在回答，因为我遇到了类似的问题，在这篇文章中我没有找到任何解决方案。 就我而言，问题是由DataFrame上的filter操作引起的。
我解决了在 DataFrame 中调用cache()的问题。
在这种情况下，一种可能的解决方案是尝试缓存 RDD：
data = spark.read.csv("datasets/mnist_test.csv").rdd\
.map(lambda x : [float(str) for str in x]).cache()\
.toDF()
features = VectorAssembler(inputCols=data.columns, outputCol="features").transform(data).select("features")
kmeans = KMeans().setK(10).setSeed(1)
model = kmeans.fit(features)
Python（pyspark）错误= ValueError：无法将字符串转换为float：“ 17”
[英]Python (pyspark) Error = ValueError: could not convert string to float: “17”
如何将“ 01MAR1978：00：00：00”字符串格式的日期转换为SparkR中的日期格式？
[英]How to Convert Date in “01MAR1978:00:00:00” string format to Date Format in SparkR?
将字符串转换为日期格式为 2020-04-21T11:28:40.321328+00:00
[英]Convert string to date with format 2020-04-21T11:28:40.321328+00:00
在Java中的Spark SQL中仅使用时间（例如：10：00：00 AM）按小时分组？
[英]group by hour using time alone (ex:10:00:00 AM) in spark sql in java?
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.
相关问题
\x00 出现在 Pyspark dataframe 中的每个字符之间
ValueError：无法将字符串转换为浮点型
无法将字符串转换为NaiveBayes Spark示例上的浮点型
Python（pyspark）错误= ValueError：无法将字符串转换为float：“ 17”
如何将“ 01MAR1978：00：00：00”字符串格式的日期转换为SparkR中的日期格式？
将字符串转换为日期格式为 2020-04-21T11:28:40.321328+00:00
每小时00:00进行Spark流运行处理
想要将字符串值转换为浮动在Spark Python中
在 Spark 1.X 中将 Excel 文件转换为 csv
在Java中的Spark SQL中仅使用时间（例如：10：00：00 AM）按小时分组？
相关标签