Heavy load?

[Palinurus]

For the first time ever when surfing to SotT I was confronted today with this message:

This website is under heavy load.

We're sorry, too many people are accessing this website at the same time. We're working on this problem. Please try again later.

Am I to conclude that SotT is becoming really popular at the moment? If so, congrats with all the new readership!

 

[mabar]

It happend to me too, I think it is great news?! :)

Edit: Add: By the way, I was trying to access Ison related news.

 

[Jacques]

It happend to me too, I think it is great news?! :)

Edit: Add: By the way, I was trying to access Ison related news.

Same thing for me.

 

[Pinocchio]

Same here... Seems to be back up now :)

 

[FrankM4326754]

Just happened for me as well.

 

[Scottie]

It is apparently the Bing search engine, which has decided that SOTT is popular. So, it starts indexing every page on the site - with no throttling.

I've told Bing to take a valium... We'll see if it works.

:P

 

[Gawan]

It is apparently the Bing search engine, which has decided that SOTT is popular. So, it starts indexing every page on the site - with no throttling.

I've told Bing to take a valium... We'll see if it works.

:P

:D Then Mr. Bing visited some time ago too. Anyway, good to know.

 

[Keit]

It is apparently the Bing search engine, which has decided that SOTT is popular. So, it starts indexing every page on the site - with no throttling.

I think Bing can see the future. ;)

 

[Palinurus]

It is apparently the Bing search engine, which has decided that SOTT is popular. So, it starts indexing every page on the site - with no throttling.

I've told Bing to take a valium... We'll see if it works.

:P

Alas, too good to be true then. Thanks for answering, Mr. Scott.

It is apparently the Bing search engine, which has decided that SOTT is popular. So, it starts indexing every page on the site - with no throttling.

I think Bing can see the future. ;)

Let us hope so. I have no objections to a future like that...

 

[Scottie]

It is apparently the Bing search engine, which has decided that SOTT is popular. So, it starts indexing every page on the site - with no throttling.

I've told Bing to take a valium... We'll see if it works.

:P

Alas, too good to be true then. Thanks for answering, Mr. Scott.

Well, sort of. I totally forgot that ages ago, I set up Bing Webmaster Tools for SOTT. This lets us see/control Bing's crawling of SOTT.

So, it appears that in the last few weeks, Bing has gone completely bonkers, and is attempting to index over 200,000 pages on SOTT every day - as fast as it can (or more accurately, as fast as SOTT can provide those pages)!

But, for example, there are 85,000 English articles on SOTT. Then, add in the old archived SOTT pages, the Category List pages, etc, and you still would have a hard time reaching 200,000+ pages.

Thus, it seems that Bing has decided that since we have content people are sharing, it better index the crap out of our site several times a day. That's not a bad thing, I guess.

Still, I told it to hold its horses. Now, it should index about 15,000 pages a day maximum. Since most of the older articles/pages on SOTT don't change much day to day (even considering Reader Comments), that should do nicely.

We'll see!

 

[Palinurus]

Interesting background info. Thanks for sharing Mr. Scott