Linux脚本返回网页上的域

Question

I was tasked with this question: Write a bash script that takes a URL as its first argument and prints out statistics of the number of links per host/domain in the HTML of the URL.

So for instance given a URL like www.bbc.co.uk it might print something like

www.bbc.co.uk: 45
bbc.com: 1
google.com: 2
Facebook.com: 4

That is, it should analyse the HTML of the page, pull out all the links, examine the href attribute, decide which links are to the same domain (figure that one out of course), and which are foreign, then produce statistics for the local ones and for the remote ones.

Rules: You may use any set of standard Linux commands in your script. You may not use any higher-level programming languages such as C or Python or Perl. You may however use awk, sed, etc.

I came up with the solution as follows:

#!/bin/sh

echo "Enter a url eg www.bbc.com:"
read url
content=$(wget "$url" -q -O -)
echo "Enter file name to store URL output"
read file
echo $content > $file
echo "Enter file name to store filtered links:"
read links
found=$(cat $file | grep -o -E 'href="([^"#]+)"' | cut -d'"' -f2 | sort | uniq | awk '/http/' > $links)
output=$(egrep -o '^http://[^/]+/' $links | sort | uniq -c > out)
cat out

I was then told that "i must look at the data, and then check that your program deals satisfactorily with all the scenarios.This reports URLs but no the domains" Is there someone out there that can help me or point me in the right direction so as i can be able to achieve my goal? what am i missing or what is the script not doing? I thought i had made it work as required.

Answer 1

The output of your script is:

      7 http://news.bbc.co.uk/
      1 http://newsvote.bbc.co.uk/
      1 http://purl.org/
      8 http://static.bbci.co.uk/
      1 http://www.bbcamerica.com/
     23 http://www.bbc.com/
    179 http://www.bbc.co.uk/
      1 http://www.bbcknowledge.com/
      1 http://www.browserchoice.eu/

I think they mean that it should look more like:

      7 news.bbc.co.uk
      1 newsvote.bbc.co.uk
      1 purl.org
      8 static.bbci.co.uk
      1 www.bbcamerica.com
     23 www.bbc.com
    179 www.bbc.co.uk
      1 www.bbcknowledge.com
      1 www.browserchoice.eu