LWP :: Simple和LWP :: UserAgent不会返回完整的网页内容

Question

I have an internal local machine, hosting analytics which I need to process. The analytics are returned in JSON format if you have the URL. for instance,

http://1.2.3.4:8081/analytics/mydata/myNodeData?flat

it would give the JSON file. Now I am getting the JSON file via LWP::Simple (also tried LWP::UserAgent) into a variable, and later parsing it. It works fine for most of my URLs. The problem is that with some of the URLs, it doesn't read the whole data into my variable, but only partial.

use LWP::Simple;                # From CPAN
use LWP::UserAgent;
use JSON qw( decode_json );     # From CPAN
use Data::Dumper;               # Perl core module
use Data::Diver qw{ Dive };         # for diving in the Hash
use strict;                     # Good practice
use warnings;                   # Good practice

#using LWP Simple    
my $trendsurl = 'http://1.2.3.4:8081/analytics/mydata/myNodeData?flat';
my $json = get( $trendsurl );
die "Could not get $trendsurl!" unless defined $json;
my $decoded_json = decode_json( $json );

#using LWP::UserAgent
my $ua = LWP::UserAgent->new();
my $req = new HTTP::Request GET => $trendsurl;
my $res = $ua->request($req);
my $content = $res->content;

In case its helpful here is the display of debug info of content (the decoded_json) is the same.

  DB<1> p $content
{"NodeStatus": {"deleted": false, "disk_usage_info": [{"partition_space_availabl
e_1k": 791475728, "partition_space_used_1k": 171611096, "partition_name": "/dev/
mapper/os-root", "partition_type": "ext4"}, {"partition_space_available_1k": 151
200, "partition_space_used_1k": 39244, "partition_name": "/dev/vda3", "partition
_type": "ext2"}], "process_info": [{"process_name": "XX-api:0", "process_s
tate": "PROCESS_STATE_RUNNING", "last_stop_time": null, "start_count": 2, "core_
file_list": [], "last_start_time": "1461822985334246", "stop_count": 0, "last_ex
it_time": null, "exit_count": 0}, {"process_name": "XX-config-nodemgr", "p
rocess_state": "PROCESS_STATE_RUNNING", "last_stop_time": null, "start_count": 2
, "core_file_list": [], "last_start_time": "1461822979324868", "stop_count": 0,
"last_exit_time": null, "exit_count": 0}, {"process_name": "XX-discovery:0
", "process_state": "PROCESS_STATE_RUNNING", "last_stop_time": null, "start_coun
t": 2, "core_file_list": [], "last_start_time": "1461822983332516", "stop_count"
: 0, "last_exit_time": null, "exit_count": 0}, {"process_name": "XX-svc-m

  DB<2>

You see the log file is abruptly terminated .. If I get this via my browser,its complete ..

EDIT:

I added the following eval statement while fetching URL and then decoding JSON .. the retreival is without error but JSON decoding fails because its not a complete JSON.

eval { $decoded_json = parse_json( $json ) };
if ($@) {
    warn "func raised an exception: $@";
}
func raised an exception: JSON error at line 1: Unexpected end of input parsing 
string starting from byte 1105 at Tester.pl line 28.

EDIT 2: I just tried another way ...

#!/usr/bin/perl  

 use LWP::UserAgent;  
 use HTTP::Request;  

 my $URL = 'http://1.2.3.4:8081/analytics/mydata/myNodeData?flat';  

 my $ua = LWP::UserAgent->new();  
 my $header = HTTP::Request->new(GET => $URL);  
 my $request = HTTP::Request->new('GET', $URL, $header);  
 my $response = $ua->request($request);  

 if ($response->is_success){  
     print "URL:$URL\nHeaders:\n";  
     print $response->headers_as_string;  
 }elsif ($response->is_error){  
     print "Error:$URL\n";  
     print $response->error_as_HTML;  
 } 

and the debug shows that $response->is_success block is selected and here is the output.

URL:http://1.2.3.4:8081/analytics/mydata/myNodeData?flat
Headers:
Connection: close
Date: Tue, 10 May 2016 19:49:40 GMT
Content-Type: application/json
Client-Aborted: die
Client-Date: Tue, 10 May 2016 19:49:38 GMT
Client-Peer: 1.2.3.4:8081
Client-Response-Num: 1
Client-Transfer-Encoding: chunked
X-Died: read failed: An existing connection was forcibly closed by the remote ho
st. at C:/Perl64/lib/LWP/Protocol/http.pm line 465.

Answer 1

I use this code or something like that, this code use POST for params, i get a large JSON with this and its works well

use LWP::UserAgent;
use Data::Dumper;
use JSON;
use JSON::Parse;

my $params = 
{(
'flat' => 1,
)};

    my $server_endpoint = "http://1.2.3.4:8081/analytics/mydata/myNodeData";
    my $ua = LWP::UserAgent->new;
    my $req = HTTP::Request->new();
    my $response = $ua->post( $server_endpoint, $params );
    my $result  = $response->decoded_content();

    $result = from_json($result);
    print Dumper $result;