![](/img/trans.png)
[英]Read data from BigQuery and write it into avro file format on cloud storage
[英]GCP Dataflow- read CSV file from Storage and write into BigQuery
我在存储中有一个CSV文件,我想读取它并将其写入BigQuery表。 这是我的CSV文件,其中第一行是标题:
GroupName,Groupcode,GroupOwner,GroupCategoryID
System Administrators,sysadmin,13456,100
Independence High Teachers,HS Teachers,,101
John Glenn Middle Teachers,MS Teachers,13458,102
Liberty Elementary Teachers,Elem Teachers,13559,103
1st Grade Teachers,1stgrade,,104
2nd Grade Teachers,2nsgrade,13561,105
3rd Grade Teachers,3rdgrade,13562,106
Guidance Department,guidance,,107
Independence Math Teachers,HS Math,13660,108
Independence English Teachers,HS English,13661,109
John Glenn 8th Grade Teachers,8thgrade,,110
John Glenn 7th Grade Teachers,7thgrade,13452,111
Elementary Parents,Elem Parents,,112
Middle School Parents,MS Parents,18001,113
High School Parents,HS Parents,18002,114
这是我的代码:
public class StorgeBq {
public static class StringToRowConverter extends DoFn<String, TableRow> {
private String[] columnNames;
private boolean isFirstRow = true;
@ProcessElement
public void processElement(ProcessContext c) {
TableRow row = new TableRow();
String[] parts = c.element().split(",");
if (isFirstRow) {
columnNames = Arrays.copyOf(parts, parts.length);
isFirstRow = false;
} else {
for (int i = 0; i < parts.length; i++) {
row.set(columnNames[i], parts[i]);
}
c.output(row);
}
}
}
public static void main(String[] args) {
DataflowPipelineOptions options = PipelineOptionsFactory.fromArgs(args).withValidation()
.as(DataflowPipelineOptions.class);
options.setZone("europe-west1-c");
options.setProject("mydata-dev");
options.setRunner(DataflowRunner.class);
Pipeline p = Pipeline.create(options);
p.apply("ReadLines", TextIO.read().from("gs://mydata3-dataflow/C2ImportGroupsSample.csv"))
.apply("ConverToBqRow",ParDo.of(new StringToRowConverter()))
.apply("WriteToBq", BigQueryIO.<TableRow>writeTableRows()
.to("mydata-dev:DF_TEST.dataflow_table")
.withWriteDisposition(WriteDisposition.WRITE_APPEND)
.withCreateDisposition(CreateDisposition.CREATE_NEVER));
p.run().waitUntilFinish();
}
}
存在一些问题:1)当作业开始执行时,我看到有一个名为“ DropInputs”的进程,我的代码中尚未定义该进程! 并开始在所有任务之前运行,为什么?
2)为什么管道不以第一个任务“ ReadLines”开头? 3)在日志文件中,我看到在任务“ WriteToBq”中它尝试查找其中一个数据作为字段,例如“ 1st Year Teachers”不是字段,而是“ GroupName”的数据:
"message" : "JSON parsing error in row starting at position 0: No such field: 1st Grade Teachers.",
您的代码中有几个问题。 但是,首先,对于“ DropInputs”阶段-您可以放心地忽略它。 这是此错误报告的结果。 我仍然不明白为什么需要显示它(它也使我们的很多用户感到困惑),并且我很希望Googler能够对此加以介绍。 我认为这只是混乱。
正确,现在输入您的代码:
boolean
标志进行检查,而是每次在ParDo
检查string
值本身,例如, if (c.element.contains("GroupName") then..
withSchema(..)
。 这是我的一个公共管道中的一个示例 。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.