將RDD轉換為Spark DataFrame時出現Unicode錯誤

Question

在數據框上運行show方法時，出現以下錯誤。

Py4JJavaError: An error occurred while calling o14904.showString.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 23450.0 failed 1 times, most recent failure: Lost task 0.0 in stage 23450.0 (TID 120652, localhost): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/Users/i854319/spark2/python/lib/pyspark.zip/pyspark/worker.py", line 172, in main
    process()
  File "/Users/i854319/spark2/python/lib/pyspark.zip/pyspark/worker.py", line 167, in process
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/Users/i854319/spark2/python/lib/pyspark.zip/pyspark/serializers.py", line 263, in dump_stream
    vs = list(itertools.islice(iterator, batch))
  File "<ipython-input-8-b76896bc4e43>", line 320, in <lambda>
UnicodeEncodeError: 'ascii' codec can't encode characters in position 3-5: ordinal not in range(128)

    at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:193)
    at org.apache.spark.api.python.PythonRunner$$anon$1.next(PythonRDD.scala:156)

當我僅獲取12行時，它不會引發錯誤。

jpsa_rf.features_df.show(12)
+------------+--------------------+
|Feature_name|    Importance_value|
+------------+--------------------+
| competitive|0.019380017988201638|
|         new|0.012416277407924172|
|self-reliant|0.009044388916918005|
|     related|0.008968947484358822|
|      retail|0.008729510712416655|
|      sales,|0.007680271475590303|
|        work|0.007548541044789985|
| performance|0.007209008630295571|
|    superior|0.007065626808393139|
|     license|0.006436001036918034|
|    industry|0.006416712169788629|
|      record|0.006227581067732823|
+------------+--------------------+
only showing top 12 rows

但是當我執行.show（15）時，我得到了錯誤。

我按如下方式創建了此數據框：它基本上是要素的數據框，其特征值來自隨機森林模型

vocab=np.array(self.cvModel.bestModel.stages[3].vocabulary)
        if est_name=="rf":
            feature_importance=self.cvModel.bestModel.stages[5].featureImportances.toArray()
            argsort_feature_indices=feature_importance.argsort()[::-1]
        elif est_name=="blr":
            feature_importance=self.cvModel.bestModel.stages[5].coefficients.toArray()
            argsort_feature_indices=abs(feature_importance).argsort()[::-1]
        # Sort the features importance array in descending order and get the indices

        feature_names=vocab[argsort_feature_indices]

        self.features_df=sc.parallelize(zip(feature_names,feature_importance[argsort_feature_indices])).\
        map(lambda x: (str(x[0]),float(x[1]))).toDF(["Feature_name","Importance_value"])

Answer 1

我假設您正在使用Python2。當前的問題很可能是df.map的str(x[0])部分。 似乎x[0]指向一個unicode字符串，並且應該將str轉換為一個字節字符串。 但是，它是通過隱式假設ASCII編碼來實現的，該編碼僅適用於純英文文本。

這不是應該做的事情。

簡短的答案是：將str(x[0])更改為x[0].encode('utf-8') 。

長答案可以在例如此處或此處找到。