Using sed to Replace Portion of String Containing Slashes and Special Characters

Question

I have a config like:

{
  "taskdb": "sqlite+taskdb:///taskdb.db",
  "projectdb": "sqlite+projectdb:///projectdb.db",
  "resultdb": "sqlite+resultdb:///resultdb.db",
  "message_queue": "redis://:tFjlvTZ%V4#&YtGl7JEa2l#&@127.0.0.1:6379/0",
}

I want to change the Redis password via a shell script. Seems it usually using sed to do such work, but I find something hard to cope with.

First, I try to match redis://:tFjlvTZ%V4#&YtGl7JEa2l#&@

I have tried the following, but they don't work:

1. `sed -i 's/^redis:\/\/:.+@$/{&}/g' config.json`
2. `sed -i 's#^redis://:.+@$#{&}#g' config.json`
3. `sed -i 's/redis:\/\/:.+@/{&}/g' config.json`

What I really want to match is tFjlvTZ%V4#&YtGl7JEa2l#& and replace it with a new password. But I even can't match the row.

Answer 1

Try this:

sed -i 's#redis://:[^@]*#redis://:NEW_PASSWORD_HERE#g' config.json

If you don't want a .bak file:

sed -i '' 's#redis://:[^@]*#redis://:NEW_PASSWORD_HERE#g' config.json

We're basically searching for "any character that isn't an @ sign" after redis://: , and replacing this whole leading string with redis://:NEW_PASSWORD_HERE .

Now, if you wanted to specifically match only a line that contained message_queue , you could do something like this as well:

sed -i '/message_queue/s#redis://:[^@]*#redis://:NEW_PASSWORD_HERE#g' config.json

Edit

As mentioned in the comments, if there's an @ in the password (although I don't think the Redis connector would be able to parse it), we can use a different method of anchoring our match:

sed -i 's#redis://:.*\(@127\.0\.0\.1\)#redis://:NEW_PASSWORD_HERE@127.0.0.1#g' config.json

Or, with less redundant typing:

sed -i 's#\(redis://:\).*\(@127\.0\.0\.1\)#\1NEW_PASSWORD_HERE\2#g' config.json

\\1 and \\2 represent the two groups enclosed between \\( and \\) .

Another Edit

You mentioned in the comments that you expected sed 's/redis:\\/\\/:.+@/{&}/g' config.json to work since Regex101 shows that it will . However, Regex101 is using Perl-based regular expression engines, rather than POSIX-style regular expressions like sed uses. Additionally, without operating in "extended regex" mode, the + operator won't work in sed . This can present a challenge for portability, as the "extended regex mode" option differs on Linux ( -r ) and Mac OS X / BSD ( -E ). Unfortunately, I cannot find a regex testing tool similar to Regex101 for POSIX/ERE regular expressions.

Answer 2

On first glance I see a few problems with your expression:

The first is that you are using ^ and $ to delimit the beginning and end of the string you want to match but those operators mean that you want to match the beginning and end of a line respectively. Since your string to match is in the middle of the line you need to get rid of them.

The second is that standard sed doesn't support the + operator. If you are using GNU sed you can give it the -r option to get extended regex notation. Or you can just use the * operator which will probably be good enough for you here.

Lastly, I don't know what you're going for with the {&} in the replacement text. Shouldn't that just be the replacement password?

Answer 3

By default, sed does not recognize + as a metacharacter. If you have GNU sed , add -r to make it recognize it; with BSD sed , you'd use -E instead. (However, we can infer that you're using GNU sed since BSD sed requires a suffix for the -i option and you're not providing one.) But the .+ notation is too greedy for safety. You really want one or more 'not- @ ' characters. The g modifier is superfluous (but otherwise harmless); you've only ever got one URL on a line in the config file. That's written:

sed 's/redis:\/\/:[^@]\{1,\}@/{&}/' config.json

That should work with any version of sed . You could also use:

sed -r 's%redis://:[^@]+@%{&}%' config.json

which is specific to GNU sed because of the -r . Changing the delimiter from slash to percent is doable in all versions of sed .

Of course, if you want to replace the password component with a new value, and as long as the new value does not contain any percent symbols, you can use:

sed -r 's%(redis://:)[^@]+@%\1new-password@%' config.json

If you need to work with the password in a variable, use double quotes instead of single quotes:

sed -r "s%(redis://:)[^@]+@%\1${new_password}@%" config.json

Also note that you really shouldn't use the -i option until you've got your script working. Granted, if you work on a copy of the data file it isn't the end of the world, but it's usually better to add the overwriting only when you're confident that the script edits the file the way you want it to.