Edit a text file line by line in bash

Question

I have a requirements.txt file that has the following lines-

PyMySQL==0.9.3
botocore==1.12.196
boto3==1.9.188

I need to edit this file in-place using bash to remove all lines containing boto3 and botocore . So far, I've come up with-

while read a ; do echo ${a//boto3==1.9.188/} ; done < requirements.txt > requirements.txt.t ; mv requirements.txt{.t,}

.. which successfully removes the line containing boto3==1.9.188 . However, this version number could be anything, say 1.10.122 .

How do I generalize the script above to remove all lines containing boto3 and botocore strings? Or is there a better way to do this? Thanks.

Answer 1

grep -vE '^(botocore|boto3)\W' requirements.txt > requirements.txt.new && \
    mv requirements.txt.new requirements.txt

Explanation:

• -v tells grep to output lines that don't match the pattern, and skip lines that do match.
• -E tells grep to allow extended regular expressions, which we use for alternation.
• ^ anchors the pattern to the beginning of the line so that it won't match foobotocore==1.2.3 .
• The (x|y) construct matches either x or y . (To add a third package, just add another | to create (x|y|z) .)
• \\W in the pattern matches a "non-word character" so that the pattern won't match botocorefoo==1.2.3 .
• && only invokes the mv command if grep was successful and matched at least one line (prevent clobbering the whole file).

Answer 2

使用awk

awk '!/(botocore|boto3)/' requirements.txt > requirements.txt.t && mv requirements.txt.t requirements.txt

Answer 3

Use ed :

printf 'g/botocore/d\ng/boto3/d\nwq\n' | ed requirements.txt

ed works by reading a set of commands (terminated by newlines) from standard input and applying them to the file named by its argument. The commands should look familiar if you are familiar with sed (indeed, sed is a s tream ed itor based on ed ).

• g/botocore/d selects each line matching the regular expression botocore and applies the d (delete) command to each.
• g/boto3/d does the same for lines matching boto3 .
• wq saves the changes and quotes.

Answer 4

Use perl:

I like perl in this case because

1. You don't need to worry about file manipulation
2. You can easily make a backup of the file as well.

Command:

$ perl -ni -e '/\b(boto3|botocore)\b/ || print;' requirements.txt

Breakdown of options

• -n iterate over lines in the file
• -i edit the file in place.
• -i.orig edit the file in place and make a backup with the extension '.orig'
• -e execute the perl command

/\\b(boto3|botocore)\\b/ || print;

This reads if the line doesn't match a word with boto3 or botocore then print. The \\b indicates a word boundary.

Example:

$ cat requirements.txt
PyMySQL==0.9.3
botocore==1.12.196
boto3==1.9.188
$ perl -ni -e '/\b(boto3|botocore)\b/ || print;' requirements.txt
$ cat requirements.txt
PyMySQL==0.9.3
$

Example with making backup of file called "requirements.txt.orig"

$ cat requirements.txt
PyMySQL==0.9.3
botocore==1.12.196
boto3==1.9.188
$ perl -ni.orig -e '/\b(boto3|botocore)\b/ || print;' requirements.txt
$ cat requirements.txt
PyMySQL==0.9.3
$ cat requirements.txt.orig
PyMySQL==0.9.3
botocore==1.12.196
boto3==1.9.188