从两个数据集中提取数据

Question

请问从这两个数据库中提取数据的最佳方法是什么，我需要正面或负面的反馈，解释原因

我有两个数据库：

• A、庞大的数据集（数百万条记录）
• D、小数据集（不到一千条记录）

第一次提取

df_A = spark.read.format("csv").option("header","true").load(A)
df_D = spark.read.format("csv").option("header","true").load(D)
df_join = df_A.join(df_D, df_A.key == df_D.key) #Supposing both datasets have a column key.
result = df_join. dropDuplicates()
result_list = result.collect_list()
import csv
with open(..., 'w', newline='') as result_file:
    wr = csv.writer(result_file, quoting=csv.QUOTE_ALL)
    wr.writerow(result_list)

第二次提取

from pyspark.sql.functions import broadcast
def bigOperation(partitionData):
updatedData = []
for row in partitionData:
result_row = …. #Big line operation
updatedData.append(result_row)
return iter(updatedData)
df_A = spark.read.format("csv").option("header","true").load(A)
df_D = spark.read.format("csv").option("header","true").load(D)
df_join = df_D.join(broadcast(df_A), ["key"])
rdd_join = df_join.mapPartitions(bigOperation)
#end