Is there a version of Apache Beam that can read JSON from Google Spanner?

Question

I'm trying to run a read query on a Spanner table that has a JSON field, type code 11.

I have attempted this in python, and it's an absolute no go. apache_beam[gcp] dependency for Spanner is version 1.19.1, which does not have a type code 11, and so has no idea what Spanner is replying with. Casting the JSON to STRING also does not work, apparently it's illegal.

I'm wondering if any pythonistas have a workaround(I doubt it)

And I'm hoping someone who has used Apache Beam with Java can tell me if they have that dependency updated. I can see java-spanner is version 6.16.1, but I can't tell which version of java-spanner Apache Beam is using, and that would be a tremendous amount of work to set it up to test it. I haven't played with Java for years.

Answer 1

Answering part of your question for the workaround.

Casting the JSON to STRING also does not work, apparently it's illegal.

Based on the following link, https://cloud.google.com/spanner/docs/functions-and-operators , JSON can't be CAST to or from any other type.

To convert a STRING to a JSON, you can use PARSE_JSON:https://cloud.google.com/spanner/docs/json_functions#parse_json

To convert a JSON to a STRING, you can use TO_JSON_STRING: https://cloud.google.com/spanner/docs/json_functions#to_json_string

The following (java)dataflow templates use the same workaround for converting json to string. https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/master/src/main/java/com/google/cloud/teleport/spanner/BuildReadFromTableOperations.java#L100