如何提取之间的内容 <text></text> Perl中来自Wikipedia的巨大xml文件的标签（可以是任何XML）？

Question

How to extract content between <text></text> tags in perl from Wikipedia?

I want to process utf-8 huge file - loading into memory not possible. File contains <text>.*?</text> for each page - this is possible to be loaded into memory - and it should be load into some variable to do some further processing:

      <text xml:space="preserve">Some text without &lt; or &lt; ....
... more text ...
... more text ...</text>

Consider that text not starts line and not ends line - important content is between <text></text> . I want to extract it and improve to generate some text file for nlp machine learning.

File can be download with:

wget http://dumps.wikimedia.org/plwiki/latest/plwiki-latest-pages-articles.xml.bz2

File can be turn into stdin pipe with:

bzip2 -c -d plwiki-latest-pages-articles.xml.bz2 | perl something > data.txt

I am not very good in Perl and can not write good code. Not know how to learn matched position, to do micro state machine or to do moving window.

Any suggestion will be welcome.

Answer 1

Something like this will do it:

#!/usr/bin/perl
use strict;
use warnings;
use XML::Twig; 

sub text_handler {
    my ( $twig, $text_elt ) = @_; 
    print $text_elt -> text; 
    $twig -> purge; 
}

my $twig = XML::Twig -> new ( twig_handlers => { 'text' => \&text_handler } ) ->  parsefile ( 'your_xml');

Note the trick here is that purge which discards previously processed XML. You can probably set a purge on other elements too, if there's a lot of stuff inbetween 'text' nodes.