模拟 Bigquery 进行集成测试

Question

While other interfaces are relatively easy to mock in my Java integration tests, I couldn't find a proper way of mocking Bigquery.

One possibility is to mock the layer I wrote on top of Bigquery itself, but I prefer mocking Bigquery in a more natural way. I'm looking for a limited, lightweight implementation, which allows defining the table contents, and supports queries using the standard API. Is there such a library? If not, what alternative approaches are recommended?

Answer 1

In unit testing it is perfectly fine to mock all external dependencies, and as long you are using interfaces to abstract out access to BigQuery client, mocking should not be an issue.

With integration testing I would rather get all my 3rd parties dependencies tested to the extend an application needs it.

For instance one case would be an ETL which streams data from external sources to BigQuery, in this case an integration test needs to verify that all data is in BigQuery as expected, which means that verification stage needs to take into account repeated, and nested messages as required.

Another case would an application that runs some business SQLs, in this case you would have populate BigQuery with some test data before applicaiton run, then the applicaiton needs to publishe the SQL output either as view/new table/or stream out of data out of for verification.

There are already some libraries taking care of integration testing with datastores including BigQuery/NoSQL/SQL

They would provide an easy solution for the cases described above and full support for SQL, dynamic macro/predicate etc ....

1. Dsunit (go-lang)
2. JDsunit (java)
3. Endly (language agnostic)

See more how to use endly for ETL and BiqQuery testing

If datastore integration test library is not an option for you and you are looking for just testing BigQuery client, the good news is that the client uses REST, so using network sniffers you can easy record what is being send back and forth, then you can use it in replayer. In order to redirect BigQuery from public BG endpoints to your replayer you would use http java proxy.