AWS胶水作业在s3上的大输入csv数据失败
[英]AWS Glue job is failing for large input csv data on s3
问题描述
For small s3 input files (~10GB), glue ETL job works fine but for the larger dataset (~200GB), the job is failing. 
对于小的s3输入文件（~10GB），胶水ETL作业可以正常工作，但对于较大的数据集（~200GB），作业失败。
Adding a part of ETL code. 添加部分ETL代码。
# Converting Dynamic frame to dataframe
df = dropnullfields3.toDF()
# create new partition column
partitioned_dataframe = df.withColumn('part_date', df['timestamp_utc'].cast('date'))
# store the data in parquet format on s3
partitioned_dataframe.write.partitionBy(['part_date']).format("parquet").save(output_lg_partitioned_dir, mode="append")
Job executed for 4 hours and threw error. 工作执行了4个小时并引发了错误。
File "script_2017-11-23-15-07-32.py", line 49, in partitioned_dataframe.write.partitionBy(['part_date']).format("parquet").save(output_lg_partitioned_dir, mode="append") File "/mnt/yarn/usercache/root/appcache/application_1511449472652_0001/container_1511449472652_0001_02_000001/pyspark.zip/pyspark/sql/readwriter.py", line 550, in save File "/mnt/yarn/usercache/root/appcache/application_1511449472652_0001/container_1511449472652_0001_02_000001/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in call File "/mnt/yarn/usercache/root/appcache/application_1511449472652_0001/container_1511449472652_0001_02_000001/pyspark.zip/pyspark/sql/utils.py", line 63, in deco File "/mnt/yarn/usercache/root/appcache/application_1511449472652_0001/container_1511449472652_0001_02_000001/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o172.save.
End of LogType:stdout
I would appreciate it if you could provide any guidance to resolve this issue. 如果您能提供解决此问题的任何指导，我将不胜感激。
1 个解决方案
解决方案1
2 2018-01-10 18:20:43
You can only set configurable options like maxResultSize during context instantiation, and glue provides you with a context (from memory you can't instantiate a new context). 您只能在上下文实例化期间设置可配置选项（如maxResultSize ，而glue会为您提供上下文（从内存中无法实例化新上下文）。 I don't think you will be able to change the value of this property. 我认为您无法更改此属性的值。
You'll normally get this error if you collect results to the driver which exceed the specified size. 如果您向驱动程序收集超过指定大小的结果，通常会收到此错误。 You aren't doing that in this case so the error is confusing. 在这种情况下你不是这样做的，所以错误令人困惑。
It seems like you are spawning 3385 tasks, which are presumably related to the dates in your input file (3385 dates, ~9 years?). 看起来你正在产生3385个任务，这些任务可能与输入文件中的日期相关（3385个日期，〜9年？）。 You might try writing this file in batches, eg 您可以尝试批量编写此文件，例如
partitioned_dataframe = df.withColumn('part_date', df['timestamp_utc'].cast('date'))
for year in range(2000,2018):
partitioned_dataframe = partitioned_dateframe.where(year(part_date) = year)
partitioned_dataframe.write.partitionBy(['part_date'])
.format("parquet")
.save(output_lg_partitioned_dir, mode="append")
I haven't checked this code; 我没有检查过这段代码; you'll at least need to import pyspark.sql.functions.year for it to work. 你至少需要导入pyspark.sql.functions.year才能工作。
When I've done data processing with Glue I simply found that batching the work was more effective than trying to get large datasets be completed successfully. 当我使用Glue进行数据处理时，我发现批处理工作比尝试成功完成大型数据集更有效。 The system is good but hard to debug; 系统很好但很难调试; the stability on large data doesn't come easily. 大数据的稳定性并不容易。
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.