How to use sed command to delete lines without backup file?

Question

I have large file with size of 130GB.

# ls -lrth
-rw-------. 1 root root 129G Apr 20 04:25 syslog.log

So I need to reduce file size by deleting line which starts with "Nov 2" , So I have given the following command,

sed -i '/Nov  2/d' syslog.log

So I can't edit file using VIM editor also.

When I trigger SED command , its creating backup file also. But I don't have much space in root. Please try to give alternate solution to delete particular line from this file without increasing space in server.

Answer 1

It does not create a real backup file. sed is a stream editor. When applied to a file with option -i it will stream that file through the sed process, write the output to a new file (a temporary one), when everything is done, it will rename the new file to the original name.

(There are options to create backup files also, but you didn't give them, so I won't mention that further.)

In your case you have a very large file and don't want to create any copy, however temporary. For this you need to open the file for reading and writing at the same time, then your sed process can overwrite the original. After this, you will have to truncate the file at the end of the writing.

To demonstrate how this can be done, we first perform a test case.

Create a test file, containing lots of lines:

seq 0 999999 > x

Now, lets say we want to remove all lines containing the digit 4 :

grep -v 4 1<>x <x

This will open the file for reading and writing as STDOUT (1), and for reading as STDIN. The grep command will read all lines and will output only the lines not containing a 4 (option -v ).

This will effectively overwrite the beginning of the original file.

You will not know how long the output is, so after the output the original contents of the file will appear:

…
999991
999992
999993
999995
999996
999997
999998
999999
537824
537825
537826
537827
537828
537829
…

You can use the Unix tool truncate to shorten your file manually afterwards. In a real scenario you will have trouble finding the right spot for this, so it makes sense to count the number of bytes written (using wc ):

(Don't forget to recreate the original x for this test.)

(grep -v 4 <x | tee /dev/stderr 1<>x) |& wc -c

This will preform the step above and additionally print out the number of bytes written to the terminal, in this example case the output will be 3653658 . Now use truncate :

truncate -s 3653658 x

Now you have the result you want.

If you want to do this in a script, ie without interaction, you can use this:

length=$((grep -v 4 <x | tee /dev/stderr 1<>x) |& wc -c)
truncate -s "$length" x

I cannot guarantee that this will work for files >2GB or >4GB on your machine; depending on your operating system (32bit?) and the versions of the installed tools you might run into largefile issues. I'd perform tests with large files first (>4GB as this is typically a limit for many things) and then cross your fingers and give it a try :)

Some caveats you have to keep in mind:

• Of course, nobody is supposed to append log entries to that log file while the procedure is running.
• Also, any abort during the running of the process (power failure, signal caught, etc.) will leave the file in an undefined state. But re-running the command again after such a mishap will in most cases produce the correct output; some lines might be doubled, but not more than a single line should be corrupted then.
• The output must be smaller than the input, of course, otherwise the writing will overtake the reading, corrupting the whole result so that lines which should be there will be missing (or truncated at the start).