pyspark：创建多于 1 个 dataframe 失败

Question

我想将几个大型 Pandas 数据帧转换为 Spark 数据帧，然后对其进行操作和合并，如下所示：

import pandas as pd
from pyspark import SparkContext,SQLContext

df1 = pd.read_csv('data1.cat',delim_whitespace=True)
df2 = pd.read_csv('data2.cat',delim_whitespace=True)

sc = SparkContext()
sql = SQLContext(sc)
spark_df1 = sql.createDataFrame(df1)
spark_df2 = sql.createDataFrame(df2)

但是出了点问题，我收到以下错误：

  File "/home/user/anaconda3/envs/conda_py3.6.8/lib/python3.6/site-packages/pyspark/sql/context.py", line 307, in createDataFrame
    return self.sparkSession.createDataFrame(data, schema, samplingRatio, verifySchema)
  File "/home/user/anaconda3/envs/conda_py3.6.8/lib/python3.6/site-packages/pyspark/sql/session.py", line 724, in createDataFrame
    data = self._convert_from_pandas(data, schema, timezone)
  File "/home/user/anaconda3/envs/conda_py3.6.8/lib/python3.6/site-packages/pyspark/sql/session.py", line 487, in _convert_from_pandas
    np_records = pdf.to_records(index=False)
  File "/home/user/anaconda3/envs/conda_py3.6.8/lib/python3.6/site-packages/pandas/core/frame.py", line 1839, in to_records
    return np.rec.fromarrays(arrays, dtype={"names": names, "formats": formats})
  File "/home/user/.local/lib/python3.6/site-packages/numpy/core/records.py", line 617, in fromarrays
    descr = sb.dtype(dtype)
ValueError: name already used as a name or title

是否可以像这样在同一个 session 中创建多个 Spark 数据帧？

Answer 1

抛出错误是因为您的 pandas DataFrame （ df1的df2 ）具有 2 个或更多列的相同列名。 更改列名称或删除它。

谢谢，萨加尔