Web Bot

Hello all,

Sorry for the multiple questions. I am fairly new to this language. Is it possible to automate submiting a form in vb.

At the moment I am getting a URL and I am traversing the files to follow links to other pages. The problem I am getting is that when i get to a page that submits a form when a button is clicked, I cannot follow this link.

If this is not possible I can get the user to manually process these pages. the problem I have then is that the session is created for the user and not the web-bot. Can a web-bot create a session?

Does anyone even know what I am talking about?

Cheers,

Padraic
.

Well, I don't, sorry

From a VB6 program you can use the webbrowser control for http, and you also can use the xmlHTTP object to "talk" to web servers.

Ok, I will explain myself better.

I am trying to create a program that will traverse a website by means of links. When I find a link i.e. <a href="">, I open this page and continue.

I am using the web transfer control to open a URL and get the source code.

The problem I am having is that to get to certain parts of the site I need to login. My question is whether it is possible to complete the login (i know the password) with my web bot.
i.e.
<code>
<form method="POST" action="page.asp">
<input type="text" id="user">
<input type="password" id="pass">
<input type="submit">
</form>
</code>

This is just a quick example of what the web bot is trying to automate.

Since I am not sure if this is possible I have another idea which might work.

Whenever these forms are encountered I could show the page to the user by means of the webbrowser control. They could complete the login manually.
The only problem then is that the session is created for the user and when I try and continue with the web bot it tells me that I have to log in.

Does that make more sense or have I made things worse?
Any answers at all are most welcome.

Cheers,
Padraic

Hmm, not recommended. Try using that object with an intelligent web server like Google and you won't like your response. The xmlHTTP object uses its own special user-agents, which some sites (like Google, and practically all online banks) do not like.

If I were you I would learn the HTTP protocol (read up on a few of the RFCs out there. Read up on HTTP/1.0 first, then move onto 1.1), then use socket based programming to connect to the web servers and carry out whatever it is that you want to do. You will have to use this method if you want to automate form posting.

You should like the HTTP protocol, its very logical, which makes it relatively simple!

Also, by using this method you can manually play about with the Cookie headers, which means that you can pass session IDs across different sockets / connections (incase you decided to make this program multithreaded, or something of the similar!)

Hey Lozware,

Thanks very much. I will look into this area.

No problem!

The following article is a really good read, and I recommend looking at it before diving into the RFCs...
http://www.jmarshall.com/easy/http/

The basic idea behind HTTP is the following...
1) The client connects to the server
2) The client sends the request (normally GET or POST)
3) The server processes the request, and sends the response.


Important: There is no other form of chatter between the client and the host. There is simply the Request (from the client) and the Response (from the server). There are no 'Send More' chunks sent out by the client. The server will just keep on sending the response until there is nothing left to send; the client just has to sit there and receive it.

Like I said, it’s a really easy protocol to learn. You should pick it up in under an hour of reading that article. And as for sockets programming...well...it's hardly challenging now is it!

Another advantage of making your app work this way is that you will use up a lot less bandwidth. Before your app would have loaded a whole webpage (which would have included downloading Images and CSS files). And because you only wanted the URLs (which are contained inside the raw HTML file), the images and CSS files would not have been needed - thus wasting a lot of bandwidth. By using sockets programming you can design your app so that it only downloads the HTML files, and nothing else.

Hey LozWare,

Thanks again for the advice. I looked into socket programming and developed a small client server application. This was a simple client -server application which communicated by means of a Port.

I know a bit about HTTP from doing Networks in college. I understand the theory behind it but I have never done any programming with it.

Is mu understanding correct that I tell winsock to listen to a port and then request a html page through this port?

If this is correct how would I go about doing this, if not what should I be doing?

Cheers,
Padraic

No. By making your application listen on a port, you are making it wait until another program connects to it - thus making it act like a server.

What you want your program to do is to connect to a remote web server. To make a program connect to another server, use the .Connect command.

Summary:

Servers Listen. So an HTTP server (like IIS for example) would Listen on port 80 and accept inbound connections made by the clients.

Clients Connect. So an HTTP client (like FireFox for example) would use the connect command to connect to a given server which is listening on port 80.

Once your client application is connected to a webserver, you can then start transfering data.

Here is a demo application that I have written in VB6 using the Winsock control.