Recommended Posts


There are companies that have programs that visit as many of the sites on the web as they can and take digital snapshots of them. They then store these snapshots (billions of them) and use them for varying purposes. The most common purpose is to provide a search service to the public, like Google does.

The programs that scour the web taking digital snapshots are called spiders. There are a handful of spiders running all the time.


Link to comment
Share on other sites


There are also internal controls one can use, even here in the forum software. I can even block parts of the forum from spiders if I wish. But it is good to be crawled by search engine spiders and be shown on search engines. I can't imagine why anyone would have a public site and not want that.

A search engine spider should not be confused with a bot spammer. Those are the bad guys. They sign up automatically and post porn and things like that. Fortunately, with the upgrade we made in the forum software, most of the ones that were bothering us have disappeared.


Link to comment
Share on other sites

OK, time to yank the covers off our mysterious guests... I have changed the setting so you can see when the site is being crawled by search engine spiders. It is generally just a handful at any given time, not hundreds. Over 200 guests in here is a lot so you may have been in a fish tank that night. :unsure:

There are several reasons why so many guests would be listed. If people are not signed in, they will show up as guests, as will people just stopping in and just interested in reading stuff. There are also programs out there that automate forum joining and posting used by spammers and bots and although they can enter the forum, they cannot post until they join, and we have made it increasingly more difficult to automatically join the forum. We have put tighter controls on the membership validation process, such as requiring a name (We have found that most spammers or bots will put USA or another country name in that field, and that is our signal to ban them).

In the internet world, driving traffic to the site is important and I periodically submit the site to directories and search engines, which can also create a flurry of spider traffic. I hope that members will also link to our site and use the social bookmarking tools we have provided to help promote our site as well. You digg?


Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now