如何在Spark DataFrame的列值的开头和结尾处消除引号？

Question

I exported dataframe into csv format. Some of its columns datatype changed from vector to string. The column values changed from [0.350562388776,0.203056015074,-0.313145598397] to '[0.350562388776,0.203056015074,-0.313145598397]'

I tried to convert it as a vector for which i used -

from pyspark.ml.linalg import Vectors, VectorUDT
from pyspark.sql.functions import udf
list_to_vector_udf = udf(lambda l: Vectors.dense(l), VectorUDT())
vectors = df.select(
list_to_vector_udf(df["result1"]).alias("res1"),
list_to_vector_udf(df["result2"]).alias("res2")
)

The column's datatype has changed from string to vector but as i applied vectorassembler it is giving an error ValueError: could not convert string to float: [0.389866781754-0.180391363533-0.212950805169] . I searched for its solutions, got the solutions for this errors but nothing worked for me.

Answer 1

This is generally is not a good approach, nevertheless try to just eval the result (result is kind of trusted, right?)

>>> a = eval('[1,2,3'])
>>> print(a)
[1,2,3]

Be aware though, that you are probably using this lib in a wrong way.