I need some inputs on file processing in Azure Data lake storage using Power Shell.
I have a pipe Delimited input file in my ADLS Gen 1 Account.
The File content looks like below
1|2|3|a,b,
3|4|5|d,h,
I am able to remove last comma using powershell in my Local PC using below code
Get-Content $file_name | ForEach-Object {$_.TrimEnd(",") }
But when i run the same query against the same file in Azure Data lake Storage Gen 1 Account nothing happen to the data . The code i am using is
Get-AzureRmDataLakeStoreItemContent -Account $accountName -Path $myrootdir/path/test.csv| ForEach-Object {$_.TrimEnd( ",") }
One observation i have is that ForEach-Object is returning only once. That is if i print hello inside ForEach-Object loop it prints only one. But i verified that there is no new line problem by running -Head and -Tail command. I am attaching a screenshot for the same.
Can you please help me to understand what i am doing wrong here and any alternative to remove last comma in each line.
I don't think you can modify the store item directly via powershell.
The Get-AzureRmDataLakeStoreItemContent
just gets the content. (Based on my experience, if it allows you to do that, it should be a command like Set-AzureRmDataLakeStoreItemContent
or Update-AzureRmDataLakeStoreItemContent
)
The workaround is to export the file -> modify it in local -> import it again.
Update :
If I do not misunderstand your question, try the command below.
((Get-AzureRmDataLakeStoreItemContent -AccountName "joydatalake1" -Path "/sss/test.csv").ToString() -split("`r")).Trim() | ForEach-Object {$_.TrimEnd(",")}
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.