How I can use a text file in LWP?

Question

I need to send requests to an HTTP server using LWP . For example, I have a file with data, and I must send requests to server foobar.baz .

use LWP::UserAgent;

$ua = LWP::UserAgent->new;
$ua->agent("$0/0.1 " . $ua->agent);
$ua->agent("Mozilla/8.0") 

$req = HTTP::Request->new(GET => 'http://www.foobar.baz');
$req->header('Accept' => 'text/html');
$res = $ua->request($req);

How I can use file.txt in

$req = HTTP::Request->new(GET => 'http://www.foobar.baz')

for every request?

For example file.txt contains

aaaa
bbbb
cccc
dddd
eeee

I need to send a request to

aaaa.foobar.baz
bbbb.foobar.baz
cccc.foobar.baz

and so on.

How can I do it?

Answer 1

This is a very simple question, and I wonder why you can't even attempt it yourself

It's just a matter of reading the file and building the complete URL from each line of text

use strict;
use warnings 'all';

use LWP::UserAgent;

my $ua = LWP::UserAgent->new;
$ua->agent("$0/0.1 " . $ua->agent);
$ua->agent("Mozilla/8.0");

open my $fh, '<', 'file.txt' or die $!;

while ( <$fh> ) {

    next unless /\S/;

    chomp;

    my $res = $ua->get( "$_.foobar.baz" );
}

Answer 2

You might find App::SimpleScan on CPAN to be useful. I wrote it for just such an application back at Yahoo! in 2005. It handles combinatorial specifications of URLs, lets you snapshot the output, etc. Plugin-based with a fairly good set of plugins, so if it won't do exactly what you want out of the box, it shouldn't be hard for you to make it work.