使用Powershell删除巨大的（300M）文本文件的顶行

Question

The difference between this question and Remove Top Line of Text File with Powershell is that my file is huge (over 300M).

The technique I am using now is based on one of the answers to that question, however it seems inefficient for huge files. Is there a faster (and less pretty) way to do it?

Answer 1

Try using a streamreader and streamwriter. This should perform quickly.

$reader = [IO.File]::OpenText("C:\Users\Andy\Documents\input.txt")
$writer = New-Object System.IO.StreamWriter("C:\Users\Andy\Documents\output.txt")

$reader.ReadLine() > $null # Skip first line.
while ($reader.Peek() -ge 0) {
    $writer.writeline($reader.ReadLine())
}

$reader.Close()
$writer.Close()

I tested by creating a 300MB text file and used this on it. output.txt was created in 3.5 seconds :-).

Update I optimized it slightly so it doesn't doesn't evaluate an expression every iteration of the loop. Now it completes in 2.9 seconds :-)

If you don't mind memory bloat you can also do this. This is faster still but takes more memory, but less than Get-Content .

$reader = [IO.File]::OpenText("C:\Users\Andy\Documents\input.txt")
$writer = New-Object System.IO.StreamWriter("C:\Users\Andy\Documents\output.txt")

$reader.ReadLine() > $null # Skip first line.
$writer.write($reader.ReadToEnd())

$reader.Close()
$writer.Close()

Answer 2

As in Andy's answer, a StreamWriter is the way to go. However, I saw better results from Measure-Command using gc $file -ReadCount 0 than a StreamReader.

$contents=gc C:\My\File.txt -ReadCount 0
$w=New-Object System.IO.StreamWriter("C:\My\File.txt")
foreach($line in $contents){if(!$f++){continue}$w.WriteLine($line)}

This also has the added benefit of being able to read from and write to the same file.