spark sql concat 和 cast 操作导致列的空值

Question

I am doing concat and cast operation inside spark SQL query as follows:

spark.sql ("select cast(concat(df_view.col1,"  ") as Long) as new_col from df_view")

But I am getting null values in resulting DF. If I just perform the cast or concat operation, I am getting the correct results, but with both operations simultaneously, I get the null values.

Please suggest if I'm missing something in the syntax, I checked other answers but couldn't figure out the issue, also I am using only spark SQL here not DF syntax operations.

Answer 1

If you are writing the file as text then just don't cast it to Long, and preferably use pad functions to make sure you are writing the right width.

I take from the comments that your issue is fixed-width files but as a general thing it makes no sense to concat an empty space and then try to cast the result as a number. You've explicitly made it not a number before.

Ideally you deal with the file format as a file format and not by arbitrarily manipulating each field, however the latter can work if you handle each field correctly.