python-mysql in Cloud DataFlowRunner?

Question

I currently have some code that queries mysql. I'd like to run this code as part of a Apache Beam Pipeline in the DataFlowRunner. Each time I try to integrate MysqlDB the job hangs. It works fine with the DirectRunner and prior to me adding MySQLDB dependencies.

Here's my setup.py

I've added comments to the lines that appear to cause the DataFlowRunner to hang.

I've tried running the example wordcount with just the apt-get and the pypi dependency.

Expected result is to be able to add MySQL deps and still be able to run the wordcount exampe.

Update: See README for what I ended up doing.

Answer 1

For anyone else who goes down this rabbit hole just use mysql+pymysql as your URL if you're using SQLAlchemy. If you're not using ORM just use pymysql.