I want to parse text from a compressed XML feed using awk

Question

I am trying to parse <title> and <description> from the compressed XML feed at http://rss.slashdot.org/Slashdot/slashdot . I am trying to do the following

curl --silent "http://rss.slashdot.org/Slashdot/slashdot" | awk '/\btitle\b(.*?)\bdescription\b/' 

and grep -E etc., but I could not get the substrings I wanted. It always returns the entire XML as it's compressed and the data is in one line.

I was able to test my Regex string by running it in a text editor.

Appreciate your help!! Thank you!

Answer 1

Using a XML parser would help, here a test with perl and XML::Twig . Adapt it to your needs.

Content of script.pl :

#!/usr/bin/env perl

use warnings;
use strict;
use XML::Twig;

my $twig = XML::Twig->new(
    twig_handlers => {
        'title' => \&extract_text,
        'description' => \&extract_text,
    },  
)->parsefile( shift );

sub extract_text {
    my ($t, $e) = @_; 
    printf qq|%s\n=================\n|, $e->tag;
    printf qq|%s\n\n|, $e->text;
}

Run it like:

curl --silent "http://rss.slashdot.org/Slashdot/slashdot" | perl script.pl -

That yiedls something like the following for each pair title and description:

title
=================
Proof-of-Concept Port of XBMC to SDL 2.0 and Wayland

description
=================
hypnosec wrote in with news that XBMC has  ...

Answer 2

Here's an XSLT solution:

curl -s -o- http://rss.slashdot.org/Slashdot/slashdot | xsltproc slashdot.xsl -

where slashdot.xsl is

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="text" />

<xsl:variable name="newline">
<xsl:text>
</xsl:text>
</xsl:variable>

<xsl:template match="/">
    <xsl:apply-templates select='//item' />
</xsl:template>

<xsl:template match='//item'>
    <xsl:value-of select='title' /><xsl:value-of select='$newline' />
    <xsl:text>====</xsl:text><xsl:value-of select='$newline' />
    <xsl:value-of select='description' /><xsl:value-of select='$newline' />
    <xsl:value-of select='$newline' />
</xsl:template>

</xsl:stylesheet>