Problem getting html source from a protected webpage with Indy & Delphi 7

Question

Using the Indy10 components for delphi, I am grabbing the source of a webpage and displaying it in a Memo control using the following code.

procedure TForm1.Button1Click(Sender: TObject);
begin
Memo1.Text := IdHTTP1.Get(Edit1.Text);
end;

When the page I am trying to display is just a normal page (ie. no login required) the results are fine, but if I try to grab the source of a page that requires a login then the result is the memo displays the source of the login page instead of the page I requested despite the fact that I am logged in to the site in both Firefox and IE browsers.

So my question is how can I "authenticate" myself with the site using the Indy components so I get the same source that I would get if I were to view the page in my browser after logging in ?

Thx, Douglas

Answer 1

You should take some time to learn how the whole HTTP thing works, because sometimes it is more complex than what it looks. A browser does a lot more than simply issuing HTTP GET or POST request with a URL, and getting back some HTML. For example they store a lot of informations about visited sites, because the HTTP headers carry a lot of useful informations. How they do it is usually browsers specific, and not system-wide. Other browser or applications may be not able to see or use them. You have to set up the HTTP headers properly for a given site, and handle situations like authentication. Sites with authentication can use broad range of techniques to allow access, from simple login pages to HTTP authentication methods. They can redirect you to pages that handle authentication, and although it can happen trasparently for an already logged user, a browser (or an application) will see what's happening and must handle that.

Answer 2

That is because Indy only does the transport for you.

Indy does not do the login; the website does.
There are dozens of ways a website can do a login.
Most of those logins require the support of a web-browser.

So you most likely need to simulate what a web-browser does.
That includes supporting all technologies that the web-site uses for the login.
That might include Cookies, extra HTTP headers, HTML 5, JavaScript, Flash or other features.

Be prepared for a lot of work...

--jeroen