c4countdown

Interesting - just noticed "Google [Bot]" is currently browsing the forum. The git.

Yeah, we get spidered quite regularly, and phpBB cleverly recognises them, although for what purpose I'm not sure.

Charlie Reams wrote:Yeah, we get spidered quite regularly, and phpBB cleverly recognises them, although for what purpose I'm not sure.

Am I right in thinking that the contents of posts on the board are not indexed by search engines?

Matt Morrison wrote:Interesting - just noticed "Google [Bot]" is currently browsing the forum. The git.

Just seen "Yahoo [Bot]" as well

Phil Reynolds wrote: Am I right in thinking that the contents of posts on the board are not indexed by search engines?

Ah, that's what it's for. Yes, they do index the pages but not the content. Odd...

Charlie Reams wrote:Yes, they do index the pages but not the content.

Sorry, you'll have to explain this for me a bit more lucidly. In what sense does Google "index the pages"?

Phil Reynolds wrote:

Charlie Reams wrote:Yes, they do index the pages but not the content.

Sorry, you'll have to explain this for me a bit more lucidly. In what sense does Google "index the pages"?

Yeah, I realise that didn't make a lot of sense. To see what I mean, try this search; it comes up with a couple of hundred pages, but if you do something like this you'll be rather surprised to discover it returns no results at all. If you look at Google's cached versions of the pages, they just say stuff like "You are not authorised to read this forum." So you can't use Google to search this place, which annoys me a bit, because in my experience of other sites it's often better than the internal search.

Charlie Reams wrote:So you can't use Google to search this place, which annoys me a bit, because in my experience of other sites it's often better than the internal search.

Hmmm. Personally, I think I prefer it that way; even though this is a publicly readable board, I sometimes post things here that I might not if I thought they were liable to show up in some random Google search. I've often come across pages on other phpBB boards when I've been searching for stuff, so it must be configurable. Maybe you should run a poll to see what the members would prefer.

Just saw one I've never seen or heard of before "Baidu [Spider]". Apparently it's from a Chinese search engine!

Two more new ones "Majestic-12 [Bot]" and "Exabot [Bot]"!

Also saw "Alexa [Bot]" and "Ask Jeeves [Bot]" recently, and "MSN [Bot]" is quite a regular.

Another new one I have never seen before this morning "Nutch [Bot]".

Another today "Google Adsense [Bot]".

I saw "Apterous Rex [Bot]" the other day.