I need to write a.txt file from a dataframe. I read that to do that I have to have a dataframe with a single column:
I'm trying to do that like this:
dataframe = dataframe.select(concat(*dataframe.columns).alias("Data"))
But it doesn't work, I think that the unpack of the columns gives some problems. And I don't want pass explicitly all the column names. Someone has an idea? Thank you
This is the output after updating code thanks to @Jonathan Lam
dataframe.show(truncate = False)
print(*[col(column) for column in dataframe.columns])
dataframe = dataframe.select(concat(*[col(column) for column in dataframe.columns]).alias("Data"))
dataframe.show(truncate = False)
Finally I found the problem: when concat meets a 'null' value the whole value becomes null. So I have to find a way to change that
I think your approach should be correct, just test:
df = spark.createDataFrame(
[("1", "2", "3")],
schema=['col1', 'col2', 'col3']
)
df.show(3, False)
+----+----+----+
|col1|col2|col3|
+----+----+----+
|1 |2 |3 |
+----+----+----+
But I am using the pyspark api col
:
df.select(
func.concat(*[func.col(col) for col in df.columns]).alias('concat')
).show(10, False)
+------+
|concat|
+------+
|123 |
+------+
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.