How to read config files with section in bash shell

Question

I have the configuration file like this in sections

[rsync_includes]
user
data
conf


[rsync_exclude]
tmp
.pyc
*/vendor


[javascript]
utils
data

I have the patterns which i want to exlude in rsync and other configuration data in that file

Now i am confused how can i use those patterns on command line

rsync -avz --exclude-from 'content from config file rsync exclude' source/ destination/

I am not sure how can read part of config file and then use on command line

Answer 1

To use --exclude-from you will have to isolate the relevant section of the config into a temporary file. This is easy to do with a bit of sed:

tmp_file=$(mktemp)
sed -n '1,/rsync_exclude/d;/\[/,$d;/^$/d;p' config.file > $tmp_file
rsync -avz --exclude-from $tmp_file source/ destination/

I am omitting error checking and cleanup for clarity.

Note that rsync can read the exclude pattern from the stdin for an - input, so this is even shorter:

sed -n '1,/rsync_exclude/d;/\[/,$d;/^$/d;p' config.file | \
  rsync -avz --exclude-from - source/ destination/

Explanation

• The 1,/rsync_exclude/d excludes all lines up to the rsync_exclude section entry
• The /\\[/,$d excludes everything from the start of the next section to the end of the file
• The /^$/d excludes empty lines (this is optional)

All of the above extracts the relevant section from the config.

Answer 2

If your configuration file is in config.ini , then run a bash script:

rm rsync-filter
while IFS= read -r line
do
    case "$line" in
        \[rsync_includes\])  command=include ;;
        \[rsync_exclude\]) command=exclude ;;
        \[*) command= ;;
        *) [ "$command"  -a "$line" ] && echo "$command $line" >>rsync-filter
    esac
done <config.ini

After that runs, it creates rsync-filter which contains both the include and exclude rules and can be used with rsync as:

rsync -avz --filter='merge rsync-filter' source/ destination/

Separately, rsync offers the -F option which is equivalent to --filter='dir-merge /.rsync-filter' . This loads include/exclude rules from the file /source/.rsync-filter and, further, as rsync goes deeper into the directory tree, it will look for and load rules from .rsync-filter files that it finds and apply those rules to files in that directory and its subdirectories. This is a powerful way to keep and organize rsync rules.

Also, the order in which rsync reads include and exclude rules is important. With these filter files, you retain control over that order. That is an important advantage when you are trying to get rsync rules to work right.

Answer 3

I will admit that I'm not familiar with rsync, but I would format that data differently, myself.

# rsync-data-file+.txt

rsync-includes:user
rsync-includes:data
rsync-includes:conf

rsync-exclude:tmp
rsync-exclude:.pyc
rsync-exclude:\*\/vendor

javascript:utils
javascript:data

From there, you can do the following:-

#!/usr/bin/env bash
set -x

while read line
do
    if [ $(echo "${line}" | sed -n '/rsync-includes/'p) ]
    then
    parameter=$(echo "${line}" | cut -d':' -f2)
    rsync "${parameter}" (other switches here etc)
fi
done < rsync-data-file+.txt

This way you can customise your command line depending on which group the parameter belongs to; so with parameters from the javascript group, you can log the operations to a different file, for instance.

Answer 4

#!/bin/sh

typeset -A Nconfig # init array

typeset -A Oconfig # init array , u can declare multiple array for each section.s

while read line
do
    if [ "$line" = "[SECTION1]" ]
    then
        SECTION1=1
        SECTION2=0
        continue
    fi
    if [ "$line" = "[SECTION2]" ]
        then
        SECTION1=0
        SECTION2=1
        continue
        fi
    if [ "$line" = "[SECTION3]" ]
        then
        SECTION1=0
        SECTION2=0
        continue
        fi



    if [ $SECTION1= 1 ]
    then
        if echo $line | grep -F = &>/dev/null
            then
            varname=$(echo "$line" | cut -d '=' -f 1)
            echo "Novar $varname"
            Nconfig[$varname]=$(echo "$line" | cut -d '=' -f 2)
        fi
    fi
    if [ $SECTION2 = 1 ]
    then
        if echo $line | grep -F = &>/dev/null
            then
            varname=$(echo "$line" | cut -d '=' -f 1)
            Oconfig[$varname]=$(echo "$line" | cut -d '=' -f 2)
        fi
    fi


   done < Config

echo "SECTION1 FROM=${Nconfig[FROM]}"
echo "SECTION2FROM=${Oconfig[FROM]}"



[SECTION1]
FROM=abc@pqr.com
TO=abc@pqr.com
SIZE=80
THRESHOULD=60
[SECTION2]
FROM=xxxx@pqr.com
TO=xxxx@pqr.com,yyyy@pqr.com
SIZE=60
THRESHOULD=30
[SECTION3]
FROM=AAAA@pqr.com
TO=BBBB@pqr.com,yyyy@pqr.com
SIZE=60
THRESHOULD=30
LOCATION=/mnt/device/user1/