从Azure数据湖存储文件中删除行逗号结尾

Question

I need some inputs on file processing in Azure Data lake storage using Power Shell.

I have a pipe Delimited input file in my ADLS Gen 1 Account.

The File content looks like below

1|2|3|a,b,
3|4|5|d,h,

I am able to remove last comma using powershell in my Local PC using below code

Get-Content $file_name | ForEach-Object {$_.TrimEnd(",")  } 

But when i run the same query against the same file in Azure Data lake Storage Gen 1 Account nothing happen to the data . The code i am using is

Get-AzureRmDataLakeStoreItemContent -Account $accountName -Path $myrootdir/path/test.csv| ForEach-Object {$_.TrimEnd( ",")  }

One observation i have is that ForEach-Object is returning only once. That is if i print hello inside ForEach-Object loop it prints only one. But i verified that there is no new line problem by running -Head and -Tail command. I am attaching a screenshot for the same.

Can you please help me to understand what i am doing wrong here and any alternative to remove last comma in each line.

Answer 1

I don't think you can modify the store item directly via powershell.

The Get-AzureRmDataLakeStoreItemContent just gets the content. (Based on my experience, if it allows you to do that, it should be a command like Set-AzureRmDataLakeStoreItemContent or Update-AzureRmDataLakeStoreItemContent )

The workaround is to export the file -> modify it in local -> import it again.

Update :

If I do not misunderstand your question, try the command below.

((Get-AzureRmDataLakeStoreItemContent -AccountName "joydatalake1" -Path "/sss/test.csv").ToString() -split("`r")).Trim() | ForEach-Object {$_.TrimEnd(",")}