pandas 数据帧中的单独数字和分类变量

Question

I have a huge list of data in spark, and I took its headers only and saved in in the pandas dataframe.

Now I want to make different list out of it to separate categorical and numerical

df2 = df.dtypes
df3 = pd.DataFrame(df2)
print(df3)

df4= df3.filter(df3[1] = 'String')

this statemnet gives error:

SyntaxError: keyword can't be an expression

Answer 1

You don't need Pandas, use pySpark dataframe.describe() to find all numeric and string columns (this will skip columns types like date , timestamp , array , struct etc.) and then filter out StringType() columns using information from df.dtypes:

from datetime import datetime
df = spark.createDataFrame([ (1, 12.3, 1.5, 'test', 13.23, datetime(2019,9,23)) ], ['i1', 'd2', 'f3', 's4', 'd5', 'dt'])
# DataFrame[i1: bigint, d2: double, f3: double, s4: string, d5: double, dt: timestamp]

# find all numeric and string columns from df (remove the first column which is `summary`)
cols = df.limit(100).describe().columns[1:]
# ['i1', 'd2', 'f3', 's4', 'd5'] 

# get a mapping of column vs dtypes of the df:
dtype_mapping = dict(df.dtypes)
#{'d2': 'double',
# 'd5': 'double',
# 'dt': 'timestamp',
# 'f3': 'double',
# 'i1': 'bigint',
# 's4': 'string'}

# filter out string-type from cols using the above mapping:
numeric_cols = [ c for c in cols if dtype_mapping[c] != 'string' ]
# ['i1', 'd2', 'f3', 'd5']

Answer 2

Along with that necessary another '=', you are missing a few things, like the index you are trying to access is '0' not '1'. Also, there is no data type 'String' in pandas DataFrame, it is 'object'. You may try something like this:

df2 = df.dtypes
df3 = pd.DataFrame(df2)
print(df3)
df4 = df3.filter(df3.iloc[:,0] == 'object')

Answer 3

you can get non numeric columns from dataframe like this

df.loc[:,df.dtypes==np.object]