stackoom
  简体   繁体   中英

How do I extract information from a webpage using perl?

 原文  2011-09-02 12:26:30  8   2    perl/ parsing/ webpage

Question

I need to extract the largest values(number) of specific names from a webpage. consider the webpage as 

 http://earth.wifi.com/isos/preFCS5.3/upgrade/

and the URL content is 

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
 <head>
  <title>Index of /isos/preFCS5.3/upgrade</title>
 </head>
 <body>
<h1>Index of /isos/preFCS5.3/upgrade</h1>
<table><tr><th><img src="/icons/blank.gif" alt="[ICO]"></th><th><a href="?C=N;O=D">Name</a></th><th><a href="?C=M;O=A">Last modified</a></th><th><a href="?C=S;O=A">Size</a></th><th><a href="?C=D;O=A">Description</a></th></tr><tr><th colspan="5"><hr></th></tr>
<tr><td valign="top"><img src="/icons/back.gif" alt="[DIR]"></td><td><a href="/isos/preFCS5.3/">Parent Directory</a></td><td>&nbsp;</td><td align="right">  - </td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="GTP-UPG-LATEST-5.3.0.160.iso">GTP-UPG-LATEST-5.3.0.160.iso</a></td><td align="right">29-Aug-2011 16:06  </td><td align="right">804M</td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="GTP-UPG-LATEST-5.3.0.169.iso">GTP-UPG-LATEST-5.3.0.169.iso</a></td><td align="right">31-Aug-2011 16:26  </td><td align="right">804M</td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="GTP-UPG-LATEST-5.3.0.172.iso">GTP-UPG-LATEST-5.3.0.172.iso</a></td><td align="right">01-Sep-2011 16:26  </td><td align="right">804M</td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="PRE-UPG-LATEST-5.3.0.157.iso">PRE-UPG-LATEST-5.3.0.157.iso</a></td><td align="right">29-Aug-2011 16:05  </td><td align="right">1.5G</td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="PRE-UPG-LATEST-5.3.0.165.iso">PRE-UPG-LATEST-5.3.0.165.iso</a></td><td align="right">31-Aug-2011 16:26  </td><td align="right">1.5G</td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="PRE-UPG-LATEST-5.3.0.168.iso">PRE-UPG-LATEST-5.3.0.168.iso</a></td><td align="right">01-Sep-2011 16:26  </td><td align="right">1.5G</td></tr>
<tr><th colspan="5"><hr></th></tr>
</table>
<address>Apache/2.2.3 (Red Hat) Server at earth.wifi.com Port 80</address>
</body></html>

In the above source you can see 172 is the largest for GTP-UPG-LATEST-5.3.0 and 168 is the largest for PRE-UPG-LATEST-5.3.0

How can I extract these values and put it to a varialble say $gtp and $pre in perl

Thanks so much in advance 

#!/usr/bin/perl 

use strict;
use warnings;

use LWP::Simple; 

my $upgrade = 'http://earth.wifi.com/isos/preFCS5.3/upgrade/';
my $website_content = get($upgrade);
if ( $website_content =~ /href=\"PRE-UPG-LATEST-5.3.0(.*?)\.iso\"/ ) 

{

my $preversion = ${1};

print $preversion;
}

This is the code I tried with but its not getting the largest value. This is code is getting the first PRE-UPG-LATEST version value that it encounters . But I need the largest of the value

2 answers

solution1
 4 ACCPTED  2011-09-02 12:52:00

An if() executes only once. Since you want to get many, you need a loop 

while ( m//g ) {

In your data it has "UPG" but your regex has "UGP", so it won't match (you should copy/paste long strings rather than (attempt to) retype them!).

This will list the data you need, I'll leave it to you to figure out how to process it. 

while ($website_content =~ /href="((?:PRE|GTP)-UPG-LATEST-.*?)\.(\d+)\.iso"/g) {
    my($file, $version) = ($1, $2);
    print "file=$file version=$version\n";
}

solution2
 1  2011-09-02 12:52:28

I would suggest that you not only use LWP::Simple, but XML::Simple too. This will allow you to example the data as a hash of hashes. It'll be a lot easier to find the largest version.

You can't parse HTML or XML with simple regular expressions because the XML data structure is too free form. Large structures can legally be broken up on separate lines. Take a look at this example: 

<a href="http://foo.com/bar/bar/">The Foobar Page</a>

It can also be expressed as: 

<a
     href="http://foo.com/bar/bar/">
     The Foobar Page
</a>

If you were looking for a href , you'll never find it. Heck, you could even look for a\\s+href and not find it.

There might be better modules to use for parsing HTML (I found HTML::Dom ), but I've never used them and don't know which one is the best one to use.

As for finding the largest version number:

There's some difficulty because there are all sorts of strange and wacky rules with version numbering. For example: 

2.2 < 2.10

Perl has something called V-Strings , but rumor has it that they've been deprecated. If this doesn't concern you, you can use Perl::Version .

Otherwise, here's a subroutine that does version comparison. Note that I also verify that each section is an integer via the /^\\d+$/ regular expression. My subroutine can return four values:

  • 0 : Both are the same size
  • 1 : First Number is bigger
  • 2 : Second Number is bigger
  • undef : There is something wrong.

Here's the program: 

my $minVersion  = "10.3.1.3";
my $userVersion = "10.3.2";

# Create the version arrays

my $result = compare($minVersion, $userVersion);

if (not defined $results) {
    print "Non-version string detected!\n";
}
elsif ($result == 0) {
print "$minVersion and $userVersion are the same\n";
}
elsif ($result == 1) {
print "$minVersion is bigger than $userVersion\n";
}
elsif ($result == 2) {
print "$userVersion is bigger than $minVersion\n";
}
else {
print "Something is wrong\n";
}


sub compare {

my $version1 = shift;
my $version2 = shift;

my @versionList1 = split /\./, $version1;
my @versionList2 = split /\./, $version2;

my $result;

while (1) {

    # Shift off the first value for comparison
    # Returns undef if there are no more values to parse

    my $versionCompare1 = shift @versionList1;
    my $versionCompare2 = shift @versionList2;

    # If both are empty, Versions Matched

    if (not defined $versionCompare1 and not defined $versionCompare2) {
    return 0;
    }

    # If $versionCompare1 is empty $version2 is bigger
    if (not defined $versionCompare1) {
    return 2;
    }
    # If $versionCompare2 is empty $version1 is bigger
    if (not defined $versionCompare2) {
    return 1;
    }

    # Make sure both are numeric or else there's an error
    if ($versionCompare1 !~ /\^d+$/ or $versionCompare2 !~ /\^\d+$/) {
    return;
    }

    if ($versionCompare1 > $versionCompare2) {
    return 1;
    }
    if ($versionCompare2 > $versionCompare1) {
    return 2;
    }
}
}

暂无
暂无

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

Related Question how to extract specific information from html webpage using perl How can I extract information from an HTML file using Perl regular expressions? Extract information from HTML using wget and Perl using perl to extract information from text files How Do I Add A Graph to a Webpage Using Perl? Pull information from webpage using wget and Perl Script how do i parse xml webpage in perl How do I extract a date range from a csv using perl/php/grep/etc? How do i search and extract commented lines from a file using perl? How do I extract device info and mountpoints from fstab using Perl?
Related Tags
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM