Bash脚本返回域而不是URL

Question

I have this bash script that i wrote to analyse the html of any given web page. What its actually supposed to do is to return the domains on that page. Currently its returning the number of URL's on that web page.

#!/bin/sh

echo "Enter a url eg www.bbc.com:"
read url
content=$(wget "$url" -q -O -)
echo "Enter file name to store URL output"
read file
echo $content > $file
echo "Enter file name to store filtered links:"
read links
found=$(cat $file | grep -o -E 'href="([^"#]+)"' | cut -d '"' -f2 | sort | uniq | awk   '/http/' > $links)
output=$(egrep -o '^http://[^/]+/' $links | sort | uniq -c > out)
cat out

How can i get it to return the domains instead of the URL's. From my programming knowledge I know its supposed to do parsing from the right but i am a newbie at bash scripting. Can someone please help me. This is as far as I have gone.

Answer 1

I know there's a better way to do this in awk but you can do this with sed, by appending this after your awk '/http/' :

| sed -e 's;https\?://;;' | sed -e 's;/.*$;;'

Then you want to move your sort and uniq to the end of that.

So that the whole line will look like:

found=$(cat $file | grep -o -E 'href="([^"#]+)"' | cut -d '"' -f2 | awk   '/http/' | sed -e 's;https\?://;;' | sed -e 's;/.*$;;' | sort | uniq -c > out)

You can get rid of this line:

output=$(egrep -o '^http://[^/]+/' $links | sort | uniq -c > out)

Answer 2

EDIT 2: Please note, that you might want to adapt the search patterns in the sed expressions to your needs. This solution considers only http[s]?:// -protocol and www. -servers...

EDIT:
If you want count and domains:

lynx -dump -listonly http://zelleke.com | \
  sed -n '4,$ s@^.*http[s]?://\([^/]*\).*$@\1@p' | \
   sort | \
     uniq -c | \
       sed 's/www.//'

gives

2 wordpress.org
10 zelleke.com

Original Answer:

You might want to use lynx for extracting links from URL

lynx -dump -listonly http://zelleke.com

gives

# blank line at the top of the output
References

   1. http://www.zelleke.com/feed/
   2. http://www.zelleke.com/comments/feed/
   3. http://www.zelleke.com/
   4. http://www.zelleke.com/#content
   5. http://www.zelleke.com/#secondary
   6. http://www.zelleke.com/
   7. http://www.zelleke.com/wp-login.php
   8. http://www.zelleke.com/feed/
   9. http://www.zelleke.com/comments/feed/
  10. http://wordpress.org/
  11. http://www.zelleke.com/
  12. http://wordpress.org/

Based on this output you achieve desired result with:

lynx -dump -listonly http://zelleke.com | \
  sed -n '4,$ s@^.*http://\([^/]*\).*$@\1@p' | \
   sort -u | \
     sed 's/www.//'

gives

wordpress.org
zelleke.com

Answer 3

You can remove path from url with sed:

sed s@http://@@; s@/.*@@

I want to say you also, that these two lines are wrong:

found=$(cat $file | grep -o -E 'href="([^"#]+)"' | cut -d '"' -f2 | sort | uniq | awk   '/http/' > $links)
output=$(egrep -o '^http://[^/]+/' $links | sort | uniq -c > out)

You must make either redirection ( > out ), or command substitution $() , but not two thing at the same time. Because the variables will be empty in this case.

This part

content=$(wget "$url" -q -O -)
echo $content > $file

would be also better to write this way:

wget "$url" -q -O - > $file

Answer 4

you may be interested by it:

http://tools.ietf.org/html/rfc3986#appendix-B

explain the way to parse uri using regex.

so you can parse an uri from the left this way, and extract the "authority" that contains domain and subdomain names.

sed -r 's_^([^:/?#]+:)?(//([^/?#]*))?.*_\3_g';
grep -Eo '[^\.]+\.[^\.]+$' # pipe with first line, give what you need

this is interesting to:

http://www.scribd.com/doc/78502575/124/Extracting-the-Host-from-a-URL

assuming that url always begin this way

https?://(www\.)?

is really hazardous.