As many of you know, the next major release of DotNetNuke ( 4.4.0 ) is focused entirely on performance and scalability improvements. However, in order to accurately identify the areas of DotNetNuke which needed to be tuned or optimized, it was first necessary to have a solid understanding of how IIS and ASP.NET operate in Windows hosting environment. In our research we found surprisingly few resources on this subject; especially resources which presented the information in a clear and concise manner. Therefore we thought it would be useful to improve the knowledge base in this area and share our findings with the community.
First, let's cover some basic terminology:
AppDomain - an Application Domain ( ASP.NET ). A container for code and data which provides isolation for an application. This generally refers to a web site or web application. An AppDomain usually has a /bin folder where its local assemblies are located. When an AppDomain is created, all assemblies in the /bin are loaded and there is no way to granularly remove an assembly once it is loaded ( even if it is not being used by the AppDomain ).
AppPool - an Application Pool ( Windows 2003 / IIS 6.0 ). A fault tolerant mechanism for managing a group of AppDomains. Each AppPool has its own worker process ( w3wp.exe ). Once an AppDomain is loaded into an AppPool, there is no way to granularly remove it ( regardless of whether it is active or not ). An AppPool has a variety of configuration options for recycling, performance, and health which allow you to optimize your server resources.
The first thing to understand is that memory is a seriously gating resource in a Windows hosting environment. Lets say you have a server with 2GB of available memory ( this is the maximum which Windows 2003 will allocate to all user processes by default - regardless of the actual RAM capacity of your server ). And lets say you are using 5 AppPools. Guidance from Microsoft indicates you should set your memory recycling threshold at 80% of available memory ( or 800 MB – whichever is less ). So the formula in this hypothetical case would be 2GB / 5 * 80% = 320 MB. You would enter this value in the Memory recycling area of the Recycling tab for the AppPool in IIS 6.0. So now lets say that an ASP.NET application has a 10 MB working set ( I say “average” because as you load more AppDomains into an AppPool, the size of the working set decreases and begins to average out ). So this would allow you to run 32 sites in the AppPool ( 320 / 10 = 32 ) before a recycle will occur. Now these 32 sites do not necessarily represent your most active sites – they are any random site which gets a request which is being managed by the AppPool. So if we look at server density and assume that we want to host 2000 sites on a box for business/economic reasons we can calculate that we need to allocate 400 sites to each App Pool. However, based on memory constraints we already know that only 32 of these 400 sites can be active at any time ( 32 / 400 = 8% ). As soon as we exceed the memory threshold for active sites, IIS recycles the AppPool - unloading all of the App Domains. Then on the next request to the AppPool, Windows needs to first create the AppPool ( which imposes some overhead ) and then create the AppDomain ( more overhead ). So invariably there will be a significant delay on these “first requests” which can not be avoided.
So what if we modify the memory recycling threshold? Well if you allow the amount of RAM to exceed the 80% recommendation, Windows will leverage virtual memory and start to do a lot of disk paging. This is an expensive operation which will seriously affect the performance of all App Pools on the server ( it stresses the box ). So although the AppPools may not recycle as frequently, you may find that your CPU utilization or Disk I/O will spike, resulting in poor server performance.
The other scenario which triggers an AppPool recycle is when the worker process for an AppPool is idle for 20 minutes. When this occurs, similar to the memory recycling threshold, the next request needs to create the AppPool as well as the AppDomain, causing a significant impact on response time.
A common technique promoted by ASP.NET developers to keep a site "alive" is sometimes referred to as a "Keep-Alive". Essentially this is a service ( internal or external ) which constantly makes requests to an AppDomain so that it never gets recycled by an idle worker process. The effect of a "Keep-Alive" on a single AppDomain in an AppPool may not be significant. But if this technique gets overused within an AppPool, it will cause the memory recycling threshold to be regularly exceeded, causing constant recycling.
With so many factors to consider, it is imperative that an ASP.NET application strives to achieve the proper balance of response time and working set ( memory allocation ). DotNetNuke 4.4.0 will definitely deliver some significant improvements in this area.