GCP Dataflow - getting data from PubSub to report to the sentry

Question

I am wondering if it is possible to get data from pubsub and send it to sentry. I mean:

sentry.init(...., integrations=[BeamIntegrations()])
try:
  with Pipeline(optins=options) as pipeline:
  (
    pipeline
    | io.ReadFromPubSub(...) // example message {'f': 'b'}, with attribut id: id1
    | some transaformations...
expect:
  // and any of above throw an error:
  with sentry.push_scope() as scope:
    sentry.set_tag('id': id1 from pubsub message) // ?
    sentry.capture_expection
    
  

Answer 1

You can't catch errors in Beam in this way.

You have to use a dead letter queue with Beam and TupleTags .

You will have 2 sinks with this system:

• The good sink
• The bad sink

I didn't used sentry but I think it's possible to sink the a PCollection to sentry .

Example of catching errors with a library called Asgarde :

https://github.com/tosun-si/pasgarde

pip install asgarde==0.16.0

 with Pipeline(optins=options) as pipeline:
  (
    input_collection = pipeline
    | io.ReadFromPubSub(...) // example message {'f': 'b'}, with attribut id: id1

    resut = (CollectionComposer.of(input_collection)
       .map(lambda el : .....)
       .map(lambda el2 : .....)
    )

    result_outputs: PCollection[YourObjectOrDict] = result.outputs

    # Failure object contains error and input element and it's given by Asgarde library
    result_failures: PCollection[Failure] = result.failures

    result_failure | 'Send errors to Sentry' >> beam.Map(your_method_to_send_to_sentry)

If it's not possible to send errors to sentry in Beam pipeline, you can think to another solution:

• The errors can be written to Bigquery for analytics
• The errors can be loggued to Cloud Logging and then this can fire an email or Slack alerting

You can also catch error and use dead letter queue in native Beam code, I propose you an example from my Github repository:

https://github.com/tosun-si/teams-league-python-dlq-native-beam-summit/blob/main/team_league/domain_ptransform/team_stats_transform.py