使用 Azure 数据湖分析/U-SQL 拆分 1TB 文件

Question

I have a 1TB Text file which has some text data in it. I want to split it into smaller chunks of data using Azure Data Lake Analytics / U-SQL.

How can I achieve this?

Answer 1

I recommend to use ADF's Mapping Data Flow feature which is built on Spark in the Cloud

1.Create a new Data Flow

2.Add a Source file

3.Add a Sink folder by defining partitioning

4.Save the data flow

5.Then add a Execute Data Flow activity and select your new file split data flow