从pyspark中的非常大的数据框中选择随机列

Question

I have a dataframe in pyspark which has around 150 columns. These columns are obtained from joining different tables. Now my requirement is to write the dataframe to a file but in a specific order like first write 1 to 50 columns then column 90 to 110 and then column 70 and 72. That is I want to select only specific columns along with rearranging them.

I know one one of the way is to use df.select("give your column order") but in my case, the columns are very large and it is not possible to write each and every column name in 'select'.

Please tell me how can I achieve this in pyspark.

Note- I cannot provide any sample data as the number of columns is very large and the column number is the main road blocker in my case.

Answer 1

You can create list of columns programmatically

first_df.join(second_df, on-'your_condition').select([column_name for column_name in first_df.columns] + [column_name for column_name in second_df.columns])

You can select random subset of columns by using random.sample(first_df.columns, number_of_columns) function.

Hope this helps :)

Answer 2

It sounds like all that you want to do is to programmatically return the list of column names, pick out some slice or slices from that list, and then select that subset of columns in some order from your dataframe. You can do this by manipulating the list df.columns. As an example:

a=[list(range(10)),list(range(1,11)),list(range(2,12))]
df=sqlContext.createDataFrame(a,schema=['col_'+i for i in 'abcdefghij'])

df is a dataframe is with columns ['col_a', 'col_b', 'col_c', 'col_d', 'col_e', 'col_f', 'col_g', 'col_h', 'col_i', 'col_j'] . You can return that list by calling df.columns which you can slice and reorder like you would any other python list. How you do that is up to you and which columns you want to select from the df and in which order. For example:

mycolumnlist=df.columns[8:9]+df.columns[0:5]
df[mycolumnlist].show()

Returns

+-----+-----+-----+-----+-----+-----+
|col_i|col_a|col_b|col_c|col_d|col_e|
+-----+-----+-----+-----+-----+-----+
|    8|    0|    1|    2|    3|    4|
|    9|    1|    2|    3|    4|    5|
|   10|    2|    3|    4|    5|    6|
+-----+-----+-----+-----+-----+-----+