Export query from MySQL to Redshift

Question

I need daily load the result of a specific query to Redshift. I've already created a table on redshift that will support the results of this query but now i'm a little stuck since i can't find a good way to solve this.

So far i tried using python but im getting lots of headaches regarding line terminators in fields that basically store a description, and character encodings.

I know lots of programs that allow you to connect to a db and make querys also have an export option to csv but since i need to automatically do this everyday i don't think any of those would work for me.

Now i would like to know if maybe there are better suited options so i can start looking into them. Im not asking for a step by step how to but just for tools/programs/etc that i should start looking into.

Answer 1

You should look into MySQL 's stored procedures and events -- using just MySQL, you can have it generate a file every day.

You can't dynamically rename the file, or overwrite it, though, so you'd need a second job which deletes the file -- this can be done with Python.

Whether you're running Windows or Linux, you should be able to schedule a batch file or python script to execute once a day, and that would be an alternate way to do that.

Does this address your question?