如何编写（保存）包含向量列的 PySpark dataframe？

Question

I'm trying to save the PySpark dataframe after transforming it using ML Pipeline. But when I save it the weird error is triggered every time. Here are the columns of this dataframe:

And the following error occurs when I try to write the dataframe into parquet file format:

I tried to use different available winutils for Hadoop from this GitHub repository but with not much luck. Please help me in this regard. How can I save this dataframe so that I can read it in any other jupyter notebook file? Feel free to ask any questions.

Note: I also tried to save simple CSV Files which does not contain vector data but the same error persists.

Thanks

Answer 1

For spark version 3.0.0 and up you can make use of pyspark.ml.functions.vector_to_array to convert the vector types to array types and then write.

from pyspark.ml.functions import vector_to_array

df = df.withColumn("dest_fact", vector_to_array("dest_fact"))\
    .withColumn("features", vector_to_array("features"))

df.write.format("parquet").mode("overwrite").save("/file/output/path")