wget在bash中解析HTML标题失败404错误

Question

I'm building a GSA keyword list. I have a list of keywords, and the urls they are supposed to link to. I need to come up with a list of titles for the links. The best place I can come up with is the title tag in the HTML.

Given a list formatted like this:

bash,PhraseMatch,http://stackoverflow.com/questions/tagged/bash,
html,PhraseMatch,http://stackoverflow.com/questions/tagged/html,
carreers,PhraseMatch,http://careers.stackoverflow.com/faq,

I want a list like this:

bash,PhraseMatch,http://stackoverflow.com/questions/tagged/bash,Newest 'bash' Questions
html,PhraseMatch,http://stackoverflow.com/questions/tagged/html,Newest 'html' Questions
carreers,PhraseMatch,http://careers.stackoverflow.com/faq,Stack Overflow Carreers 2.0

All it is doing is looking up the URL, getting the title tag, and appending it to the end of the line. Here is what I have so far:

{
for line in $( cut -d ',' -f 3 input.csv );
{
    wget --no-check-certificate --quiet -O - $line \
    | paste -sd ' ' - \
    | grep -o -e '<head[^>]*>.*</head>' \
    | grep -o -e '<title>.*</title>' \
    | cut -d '>' -f 2 \
    | cut -d '<' -f 1 \
    | cut -d '-' -f 1 \
    | tr -d '   ' \
    | sed 's| *\(.*\)|\1|g' \
    | paste -s -d '\n' - \
    ;
}
} | paste -d ' ' input.csv - > output.csv

The problem I am having is that some of the pages are returning various errors. In that case, I get no data back. This results in no line being generated. When I do the paste to merge the two streams, they aren't the same size.

I'm looking for a way to check for empty data and return an empty line. Help?

Answer 1

Ignoring the issue parsing HTML using a collection of command-line tools, you can substitute a fixed error string for the output of commands that don't complete. ( I don't think I'm inserting the check at the right place in the pipeline, but hopefully you can make that correction):

set -o pipefail
while IFS=, read first second line rest; do
    wget --no-check-certificate --quiet -O - $line | 
      paste -sd ' ' - |
      grep -o -e '<head[^>]*>.*</head>' |
      grep -o -e '<title>.*</title>' |
      cut -d '>' -f 2 |
      cut -d '<' -f 1 |
      cut -d '-' -f 1 |
      tr -d '   ' | 
      sed 's| *\(.*\)|\1|g' | 
      paste -s -d '\n' - \
  || echo "<no output found>"   # If any part of the pipeline fails
 done < input.csv | paste -d ' ' input.csv - > output.csv