How search engines search through dynamic websites?

I apologize in advance if this section is not the right one for this kind of question but I thought it be the most relevant one.

My question is that how do search engines search though dynamic websites whose data is stored in DB. Whenever I search for some question, Yahoo Answers, Wikipedia, and many other forums appear in the Google. I myself am developing a forum so I a worried that do I have to take some steps for the data in my DB to be available for the search engine to read?

CB

all search engines are using Crawlers.

the crawler start from the website main page, where it index the contents
of that page and then it crawls to all the linked pages, meaning it go through
the links in the home page one by one and then doing the same action in
each of those pages.

for example, if inside home.asp you have link to showpost.asp?id=1
then the crawler will visit showpost.asp?id=1 and index the contents then
crawl any links in that page as well.

in forum there should be no problem: somewhere, there is link to each
and every post.

since the link URL also have impact, some websites (like this one) are using
URL-Rewrite to have "friendly" address with meaningful words instead of
plain ID numbers but it's not vital: the pages will still be indexed even
without friendly names.

Thanks for the explanation but I am still a little confused. You said that IF there is a link showpost.asp?id=1 then it would search it but in forums, there are no hardwired links. All links are generated dynamically through loops so how crawlers work with that?

CB

it's hard coded for the crawler.
the crawler won't see the source code, it will see the output.
and the output contains links.

wow thats cool. But then how do we hide the private messages?

CB

If you have proper code in place, for example:



Then the output will not be generated and the crawler will not see the page.

I wanted to comment on URL rewriting...this doesn't seem to be much of a factor anymore (at least with Google). However, it still provides users to your site a more simple URL to remember. So if you can implement it, I would.

Crawlers are not registered users.. they will "see" what anonymous visitors
are seeing, and they don't have any Private Messages.

if only registered users have link to their Inbox (private messages) then
you will have no problem.

Thank you all for your help.

CB