[英]How to find rejected files due to errors in apache beam java sdk
I Have N
number of same type files to be processed and I will be giving a wildcard input pattern( C:\\users\\*\\*
).我有N
个相同类型的文件要处理,我将给出一个通配符输入模式( C:\\users\\*\\*
)。 So now how do I find the file name and record ,that has been rejected while uploading to bigquery in java.那么现在我如何找到在java中上传到bigquery时被拒绝的文件名和记录。
I guess BQ writes to the temp location path that you pass to your pipeline and not to local [honestly not sure about this].我猜 BQ 会写入您传递给管道而不是本地的临时位置路径 [老实说,对此不确定]。
In my case, with python, I used to pass tmp location as GCS bucket, and when I error is show, they usually shows the name of the log file that contains the rejected errors in the command line logs.在我的情况下,使用 python,我曾经将 tmp 位置作为 GCS 存储桶传递,当我显示错误时,它们通常会在命令行日志中显示包含被拒绝错误的日志文件的名称。
And then I use gsutil cp
command to copy it to my local computer and read it然后我使用gsutil cp
命令将它复制到我的本地计算机并读取它
BigQuery I/O (Java and Python SDK) supports deadletter pattern: https://beam.apache.org/documentation/patterns/bigqueryio/ . BigQuery I/O(Java 和 Python SDK)支持死信模式: https ://beam.apache.org/documentation/patterns/bigqueryio/。
Java爪哇
result
.getFailedInsertsWithErr()
.apply(
MapElements.into(TypeDescriptors.strings())
.via(
x -> {
System.out.println(" The table was " + x.getTable());
System.out.println(" The row was " + x.getRow());
System.out.println(" The error was " + x.getError());
return "";
}));
Python Python
errors = (
result['FailedRows']
| 'PrintErrors' >>
beam.FlatMap(lambda err: print("Error Found {}".format(err))))
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.