Ouch. Ouch, ouch, ouch...
So what do you think of the spider traffic on www.dotnetnuke.com in the last 5 days?
If you read my previous blog regarding MS Search Robot... you already know why I'm hurting. I've been conversing via email this week with a number of good folks ranging from hosting companies on the opposite side of the planet to government agencies in my own home state of Washington. Invariably, the culprit winds up being some ill-configured instance of MS Search. Honestly, I'm not even certain of all the ways this can happen ( though I'm learning )... we know that one way is through SharePoint ( as discussed last time ).
What we DO know is that an awful lot of really nice, well meaning folks are winding up with rogue instances of MS Search eating all the bandwidth in their networks as well as putting a little dent in our own. That's 103 gigabytes ( yes, gigabytes ) of data transfer on 1.1 million page views over the course 5 days from just a couple of sources at a time. Want to know another staggering statistic?
Hourly page loads in the range of 4,900 - 197,000 by this bot alone.
Next time somebody asks you if DotNetNuke scales... you mention that number to them... which equates to 55pps (pages per second) just from this one bot, sustained steadily over a 9 hour period.
Do yourself a favor... if you're running MS Search, check your configuration. If you're a shop where anyone has the authority to configure an instance of MS Search in your environment ( anywhere in your environment ).... be monitoring for it. Because 103GB of data transfer coming out of of our pipes... is clogging up somebody elses...