XML File Using Azure Data Factory

Question

I have used the latest xml file system support feature available in the Azure data factory to convert a 2 GB xml file to csv using copy activity, but it ended up in a memory exception. But if I use a lesser sized files, eg: 500 mb one, the file gets processed within 1 hour without any parallelism a or DIU. The 2 GB file doesnt get converted even if I use all the permutation combinations of parallelism, DIU, or Block size? Am I doing something wrong. If anyone could suggest a solution, it would be really helpful.

Regards, Sandeep

Answer 1

I would recommend you to try the XML to CSV conversion using Data Flow instead if you are experiencing issues with the standard Copy activity. I would also suggest you provision a memory optimized integration runtime to execute you data flow activity.