简体   繁体   中英

Data is not flowing in PcollectionView stage in Java Apache beam

After reading data from Bigquery I have to send the data as side Input to next level.So below is the step I am following -

  1. Read BQ
  2. Convert PCollection to PCollection<KV<K,V>>
  3. Converting Pcollection to PcollectionView (Since next step with accept the PcollectionView only as side input)

But Data is not flowing to step 3.Due to this I am not able to send the data in next level where our side Input will go.

Below is the Dataflow -

在此处输入图像描述

Below is the code snippet with Java -

PCollection<TableRow> bq_row = pipeline.apply(
    "Read from BigQuery query",
    BigQueryIO.readTableRows().withoutValidation().withTemplateCompatibility().fromQuery(query).usingStandardSql());
    
PCollection<KV<String, TableRow>> rowsKeyedByUser = bq_row
    .apply(WithKeys.of(new SerializableFunction<TableRow, String>() {
        @Override
        public String apply(TableRow row) {
            return (String) row.get("asset_id");
        }
    }));
    
PCollectionView<Map<String, TableRow>> bq_row1 = rowsKeyedByUser.apply(
    "viewTags", View.<String, TableRow>asMap());

Please let us know, Is there anything missing from my end?

View operations don't really have PCollection outputs, they're just used to wire up side inputs. Have you confirmed that the side input in question is not getting any data?

If this is a streaming pipeline, do you have your windowing set up correctly? If not the side input must be computed in its entirety before the DoFn consuming it will emit anything. In particular side inputs are not available until the watermark passes the end of the corresponding window (aka we know the window has everything it will have). For the global window, that is essentially the end of time. Triggers can be used if one needs to access values of a window before it closes.

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM