使用 grep 搜索文件并且只输出部分行

Question

I'm looking though log files and trying to get a less cluttered output in my end file. If I grep for a value I want to then format the output to remove anything but the date and the url.

For example here is a line of the file.

Sep 25 08:07:51 10.20.30.40 FF_STUFF[]: 1545324890 1 55.44.33.22 10.9.8.7 - 10.60.154.41 http://website.com 0 BYF ALLOWED CLEAN 2 1 0 0 0 (-) 0 - 0 - 0 sqm.microsoft.com - [-] sqm.microsoft.com - - 0

I want to do a grep, or a better command if necessary, to output to a .txt file with only the bold entries listed. Basically list date and URL. So how do I tell it to list the first 15 characters including spaces, then find the first http/https and list everything until the first empty space? Each line is not the same length or anything of that nature so I can not just go by character position.

So my output would be

Sep 25 08:07:51 http://website.com

Thank you.

Answer 1

You can't easily use -o option in grep because you have two patterns, separated by a variable number of characters (and -o will print the complete matched part).

If you wanted to extract only URLs, this would suffice:

$ grep -oE 'https?:[^ ]+' file
http://website.com

But to extract both the date and the URL, probably the simplest solution is with GNU awk :

$ awk '{ match($0, /https?:[^ ]+/, url); print $1, $2, $3, url[0]; }' file
Sep 25 08:07:51 http://website.com

where you print first three fields ( $1 to $3 , space-separated), then search for a URL with match() (assuming it contains no spaces, ie that space characters are always properly escaped; either as + , or %20 ), and then print the first URL found (after the date).

In case you have POSIX awk (or call gawk with --posix flag), the solution is a little bit more verbose, since POSIX match() doesn't support saving of the matched parts into an array (third argument, url ) and you'll have to explicitly extract URL with substr() when a match is found:

$ awk '{ match($0, /https?:[^ ]+/); print $1, $2, $3, substr($0, RSTART, RLENGTH); }' file
Sep 25 08:07:51 http://website.com

Answer 2

To supplement @randomir's answer, we can also use sed :

$ sed 's/\(.\{15\}\).*\(https\?:\/\/[^ ]\+\).*/\1 \2/' < input.txt > output.txt

This pattern assumes that the first 15 characters composes the date and that the URL contains no spaces. It works for both http and https URLs.

---

Edit - to address the comment—for the sake of learning, we can also invoke sed to perform line-matching operations like grep :

sed -n '/10\.45\.19\.151/p' < input.txt

...will output any lines in input.txt that contain the IP address 10.45.19.151 . The -n option suppresses output of every line. We combine this option with the p command to print only lines that match the pattern.

We can merge this approach with the first command to "grep" for lines and transform them using a single command:

sed -n '/<line-match-pattern>/ s/<...>/<...>/ p' < input.txt

...will select only the lines that match <line-match-pattern> , perform the substitution, and output the result. To illustrate, here's an example using the information provided in the comment:

sed -n '/10\.45\.19\.151/ s/\(.\{15\}\).*\(https\?:\/\/[^ ]\+\).*/\1 \2/ p' \
    < messages-20171001 \
    > /backup/mikesanders-fwlog-10012017.txt

Answer 3

awk '{match($0,/http[^com]*/);print $1,$2,$3,substr($0,RSTART,RLENGTH+3)}'  Input_file

Explanation of above code:

awk '{
match($0,/http[^com]*/);                  ##Using match default utility of awk where I am searching for regex where it will look for string http till string com comes.
print $1,$2,$3,substr($0,RSTART,RLENGTH+3)##Now printing the 1st, 2nd and 3rd column which are date and time in current line and printing sub string of current line where it should start substring from the value of RSTART till value of RLENGTH(which will be http complete URL actually). Now point to be noted here variables RSTART and RLENGTH are default variables of awk which will be set once a regex match is found in match utility of awk.
}
' Input_file                              ##Mentioning the Input_file name here.

Answer 4

You can use grep -o to match each of the line sections that you want, then reassemble the lines that grep returns:

$ grep -Eo '^.{15}|https?://[^ ]+' f | paste - -
Sep 25 08:07:51 http://website.com

Note that in FreeBSD or OSX, the old version of GNU grep they use (2.5.1) is buggy, so more explicit date recognition is in order:

$ grep -Eo '[A-Z][a-z]{2} ([0-9]{2}[ :]){3}[0-9]{2}|https?://[^ ]+' f | paste - -
Sep 25 08:07:51 http://website.com

A workaround in FreeBSD is to use bsdgrep , which is functionally equivalent to gnu grep but without the bugs. In MacOS, one might need to install an alternative using homebrew or macports .. or just use the POSIX awk solution in another answer.

Anyway, in both cases, the regular expression consists of two expressions joined with an or-bar ( | , before https ). The first subexpression matches your dates, the second one matches your URLs.

As long as every line of input contains text that match both of these elements, you should get two lines of output from grep for each log entry. Then paste reassembles them into a single line.

Answer 5

Just 1 command line like:

msr -p my.log -t "^(.*?\\d+:\\d+:\\d+).*?(https?://\\S+).*" -o '$1 $2' -PIC > output.txt

• If first 15 characters is more reliable than pattern "^(.*?\\d+:\\d+:\\d+)" :

  Use "^(.{15})" like: -t "^(.{15}).*?(https?://\\S+).*"

• If you want to further filter like containing one ip 10.9.8.7 as a plaint-text( -x ):

  msr -p my.log -x 10.9.8.7 -t "^(.*?\\d+:\\d+:\\d+).*?(https?://\\S+).*" -o '$1 $2'

• If must contain more IPs like 10.9.8.7 10.9.8.8 10.9.8.9 , or further processing:

  msr -p my.log -t "^(.*?\\d+:\\d+:\\d+).*?(https?://\\S+).*" -o '$1 $2' -PAC | msr -t "10\\.9\\.8\\.[7-9]" -PAC > output.txt

msr.exe / msr.gcc* is a single exe tool for such ETL alike work (Load -> Extract -> Transform or Replace file) in my open project , about 1.6MB, no dependencies, with cross platform versions plus x86 / x64 versions.

• Load files recursively ( -r ) and filter directory name, file name, time, size like:

  -r -p dir1,dirN,file1,fileN -f "\\.(log|txt)$" --w1 2017-09-25 and --nf "excluded-files" --nd "excluded-directories" , --s1 1.5MB --s2 30MB , --w2 "2017-09-30 22:30:50" etc.

• Extract by general Regex unlike sed or awk , exactly same as C++ / C# / Java / Scala /etc.:

  -t "^(.*?\\d+:\\d+:\\d+).*?(https?://\\S+).*" ignore case: add -i like: -i -t or -it

• Transform output like:

  • -o '$1 $2' for Linux or Cygwin / Powershell on Windows.
  • -o "$1 $2" for Windows CMD console window or *.bat / *.cmd files.

See following screenshot:

If you're on Linux, you can just run msr.gcc48 or msr-i386.gcc48 it's 32-bit machine. Just run the exe you'll get the usages and examples, or see online docs about performance comparison (with Linux system tool grep and Windows system tool findstr ), built-in docs like:msr on CentOS , colorful vivid demo on Windows .