无论使用Python还是Java，读取网页都会丢失一些信息

Question

When I try to use read some web page, the page source looks like as following:

<p/><table border="1" align="center" cellpadding="10"><tbody><tr><td><a href="/cgi-bin/query/C?c101:./temp/~c1011jI5AQ" title="Displays without navigation or highlighting">Printer Friendly</a>[<a href="/home/billdwnloadhelp.html">Help</a>]</td>

But when I use Python's urllib2, urllib or requests to read this web page, the results are the same as following:

<p/><a href="/[<a href="%s">Help</a>]</td>`/C?query:c101" Printer Friendly</a><p/>

So, why I cannot read all the information and lost the very important part C?c101:./temp/~c1011jI5AQ ???

I tried to use Java to read, it is the same situation. And I try to use different OS, like Mac, Linux or Windows, it is also the same result. So how can I solve this problem?

Answer 1

I'm not sure to understand correctly: the second example is what you get either with Python or Java. And the first ? Is it obtained by looking at "source code" with a browser? In that case, three possible scenarios:

• First (and less likely), the "view source code" of your browser display source modified/altered/generated by JavaScript
• Second, the server generate different content based on the "client signature" (formally, user-agent identification )
• Third, the server provide different content based on the cookies stored on your browser

As a test, you might use curl to download page and do some comparisons -- it will be perfectly suited for that task since, among many options, you have the opportunity to change the user-agent identification provided to the server -- and so, pretending to be IE or Firefox or whatever you like:

curl(1)                           Curl Manual                          curl(1)

NAME
       curl - transfer a URL

SYNOPSIS
       curl [options] [URL...]

DESCRIPTION
       curl  is  a tool to transfer data from or to a server, using one of the
       supported protocols (HTTP, HTTPS, FTP, FTPS,  SCP,  SFTP,  TFTP,  DICT,
       TELNET,  LDAP  or  FILE).  The command is designed to work without user
       interaction.

       curl offers a busload of useful tricks like proxy support, user authen‐
       tication,  FTP upload, HTTP post, SSL connections, cookies, file trans‐
       fer resume and more. As you will see below, the number of features will
       make your head spin!

[...]

      -A/--user-agent 
              (HTTP) Specify the User-Agent string to send to the HTTP server.
              Some   badly   done  CGIs  fail  if  this  field  isn't  set  to
              "Mozilla/4.0". To encode blanks  in  the  string,  surround  the
              string  with  single  quote marks. This can also be set with the
              -H/--header option of course.

              If this option is set more than once, the last one will  be  the
              one that's used.