LWP :: Simple get的结果被截断

Question

I'm using perl to perform a get request on a url, and the results appear to be truncated.

If I run

curl myurl | wc -l

the result is 1823, and if I create the following file foo.pl:

#!/usr/bin/perl

my $url = 'myurl';

use LWP::Simple;
my $content = get $url;
die "Couldn't get $url" unless defined $content;
print $content;

and run

./foo.pl | wc -l

the result varies from around 1300 to occasionally 1823. Manually inspecting the output shows that the output is broken mid-line when using perl.

What can be causing this?

Answer 1

What happens if you turn buffering off? I also agree with Karsten S. in checking the http headers for erroneous codes. Finally, I'd also try storing the contents into an array just to see what happens.

To turn off buffering, you could simply place a $|++ at the top of your script after your use statements. Again, a shot in the dark.

To examine the http headers, you can use CGI . Here's a little site with a good example on how to get the headers from the request:

http://www.velocityreviews.com/forums/t24118-re-lwp-simple-header-information-problems.html

Finally, try using an array, @contents , to store the contents from the webserver instead of a scalar, $contents . I've had times in the past where something is passed from the remote server that Perl misinterprets as a list. I'm not sure if LWP::Simple accounts for these times but it can't hurt to try. You might only be getting one part of the data and the rest is either getting overwritten or ignored altogether. Placing the data into an array could help determine if that is happening.