OLEDB 或 ODBC 直接导入查询时出错

Question

I have a query that has around 160 million rows that would take me 4 hours to import into Power BI.

Ever since my company changed their server to Azure, I can never import this query successfully. It would start loading for 1 million rows-ish, after 1 minute or two this error always pops out.

I tried:

1. changing the command time out to 200 minutes, still errors out within loading for a minute or two, sometimes within 10 seconds
2. if I select top 1000 rows in my query, it will complete without error. But when I switch back to the original query, it always fails.

Attaching the error message. I have talked to the DE in my team and they don't seem to have a clue. Does anyone have any idea on how to fix this?

this is the error message

Answer 1

We had the same problem and went for a delta lake in combination with option 1.

1. You have to ask your self first why you are importing so much data. Always keep your model as small as possible. I can't imagine you are looking at every row. If this is needed you could you could use a combination of direct query for details and loading an aggregate for your reporting. But load aggregates instead of everything.
2. Maybe your can load less history, like the last two years.
3. You could look into loading incrementally, you could load per partition.
4. You can try to increase the DTU's for your server.