There is never a dull moment when it comes to DotNetNuke, and the past week certainly had more than its fair share of excitement. The combination of a new product release, infrastructure upgrades, and broad media exposure had a noticeable impact on the ecosystem - and resulted in some unexpected challenges. The most visible issues manifested themselves in the focal point of the DotNetNuke community - the dotnetnuke.com website. So if you count yourselves amongst the thousands who visited the website last week and experienced uncharacteristically slow response times, we sincerely apologize for the inconvenience. The good news is the issues have been resolved and the website is now performing better than ever. I would like to provide some insight into what happened.
As per our standard development practice, extensive testing of our product is conducted internally before the bits are ever made available for public consumption. Once we are satisfied with internal testing, we typically make Beta builds available to select individuals in the community who are interested in assisting us with validating the software. With the DNN 5.1 release, we went one step farther and opened up the Beta process to the entire community. We released 7 public Beta packages over the past month as we progressively moved the product closer to final release. While Beta testing was ongoing, we were also testing our own upgrade experience continuously by refreshing a staging environment with production data and content and running through the upgrade process.
Last weekend we decided the software was ready for production deployment - a term we call “dogfooding” (i.e.,eating our own dog food). We successfully upgraded dotnetnuke.com and everything seemed to behave fine on Saturday and Sunday, leading us to believe we were on track for a product release on June 16th. But on Monday morning the situation changed...
Before going any further, let me explain a little about our production environment. From humble beginnings as a bootstrapped venture, DotNetNuke Corporation has always been privileged to have MaximumASP as a project sponsor. MaximumASP has provided servers and bandwidth to the project for many years and we are deeply appreciative of their ongoing support. That being said, the servers we were using in our production environment (i.e., 32 bit, dual core CPU, 2 GB RAM ) had not kept pace with the growth of the DotNetNuke project. We recognized that at some point we were going to need to update them. We had already experienced some performance issues in the past but had always managed to work through them with a combination of configuration and software optimizations. But eventually you hit a ceiling.
On Monday and Tuesday we began to notice some serious performance issues with the dotnetnuke.com website. The CPU usage on the 2 web servers spiked above 90% and did not come down. Page loads were taking upwards of 4 to 5 seconds for unauthenticated users and authenticated usage was almost impossible. Symptoms pointed to a bottleneck somewhere and further diagnosis seemed to point to a RAM deficiency resulting in excessive page file thrash. But what happened on Monday and Tuesday to trigger this behavior?
Recently we had forged a business relationship with a professional public relations firm, Eastwick Communications. The plan was to conduct a broad media outreach program to help raise awareness of DotNetNuke in a variety of traditional business channels. Over the past couple weeks we conducted interviews with a variety of notable media organizations covering our product space, including CMSWatch, CMSWire, Redmond Developer News, Webhost Industry Review, Windows IT Pro, etc... We also met with analysts from both Gartner and Forrester to ensure growth of the DotNetNuke ecosystem was not going unnoticed. All of the extra media attention resulted in a spike in visitors to our website and our website unfortunately could not keep up with the demand.
To deal with the performance problem, we took a two pronged approach. While the Engineering team analyzed the software for performance bottlenecks in the application logic and database layer, we also reached out to MaximumASP to get new modern web servers. Scott Willhite was instrumental in getting the new servers (64 bit, quad core, 8 GB RAM) configured in record time (with assistance from Microsoft Professional Support and MaximumASP). After deploying a DotNetNuke 5.1 package with some performance optimizations Thursday night, the health of the site was back under control and the system was serving pages faster than ever before.
So what about the 5.1 release? With the performance issues resolved, we are now gearing up for the final release of both the 5.1 Professional and Community Editions. With a successful dogfooding exercise under our belt, I am confident the 5.1 platform is ready for primetime deployments. The launch will likely occur early this week.
Going forward, there are some lessons learned which we will incorporate into our process. For the early detection and resolution of performance issues, we are going to invest in a serious load testing environment where we can push the software to its limits in a controlled, methodical manner. This should mitigate some performance problems during the development process; however, it will not completely emulate the demands of a real production environment. So we will continue to eat our own dogfood in the future, as we feel it is a great benefit to our users and customers to know that we utilize the latest version of our own platform. Our ongoing commitment to the community is to ensure we provide great value in terms of a world class, enterprise-class software platform and project infrastructure.
We would like to thank everyone in the DotNetNuke community for their patience last week, as we worked through issues with the website and validating the software for production use. I must also thank the members of the DotNetNuke Corporation Engineering team (i.e., Charles, Joe, Scott) for pulling a couple of all-nighters as they worked hard to diagnose and correct the problems. And I also want to thank MaximumASP for all their support and for helping us get the new web servers added to our environment so quickly and professionally.
I’m looking forward to a highly successful launch of DotNetNuke 5.1!