如何使用Logic Apps将巨大表从Azure SQL数据库导出到CSV

Question

I work on proof of concept of Azure Logic Apps. I have a table with 70 columns and 10 million rows. I use Logic Apps to export data and upload to FTP.

The process works on a small dataset (3000 rows); however, the process fails on the whole table.

I get the error message below:

BadRequest. Http request failed as there is an error: 'Cannot write more bytes to the buffer than the configured maximum buffer size: 104857600.'

Logic App Steps:

1. Recurrence
2. SELECT statement - failure
3. Create CSV Table
4. Upload to FTP

Please advise. I could use a stored procedure and batch it in 5,000 rows but I'm not sure how to implement it in Logic Apps.

Answer 1

Based on this documentation, there is a file a HTTP message size limit of 100 MB and a file size limit of 50 MB when processing FTP files. You can use chunking to workaround those limits, if the connector supports it.