So now that the site has launched, I can announce that the hybrid solution was for 1.FM. First off, let me mention that DNN is "heavy." I hate that it's a website and not a web application. The build times are incredible. I'd much rather build locally then upload and have it just work. Instead, after you upload a file, it reocmpiles the whole application. With an application as big as 1.FM, this can take a minute or two, even on their new 8-core server with 8GB of RAM! The one time I thought IIS went down because it took so long.
However, with that said, I still say the benefits of the solution far outweigh the hurdles. First of all, I get a prepackaged set of modules for purchase at www.snowcovered.com. Second of all, it gives me a single authentication and authorization system for anything that I wanted to custom build. Combined, there isn't much I can't dothat would be within 1.FM's needs.
At launch, we were faced with two performance issues that I think are worth mentioning. The first is something to be ever aware of when using prepackaged software: just because it looks like it does what you want, doesn't mean it does it well! Most of the features that are on every page and that deal directly with 1.FM data I built myself and made heavy usage of data caching. However, when we launched, there was something on the site that was just killing us and more or less bringing the site to a standstill (with about 4100 concurrent users). After profiling with RedGate's Ants Profiler we found that it was the "latest videos" module on the homepage. Apaprently it wasn't caching any of its data. And of course it was the one boxed module I was relying on to be on every page! We solved the issue by wrapping the module control in our own custom User Control and using OutputCache. Since there was no need to ever hide videos -- all videos are public all the time -- there was no reason for us not to cache the rendered control itself.
And this brings us to the second issue. I completely underestimated the power of caching controls! I was very strict about the way that I cached the data. Everything was cached and fetched in an application thread, so no user felt the burden of requesting data ever. However, with 4000 people hitting the site at the same time, just databinding and rendering that data became a HUGE bottleneck. We were maxing out the 8 CPU cores consistently. With just a few strategically placed OutputCache's in my user controls (actually I think it only took _three_) we were able to keep the server load at approximately 20-30% CPU usage with that server load, which is more than acceptable. I have to give all the credit to Phil for this one. He stepped away from his project of the day to run the profiler and tell me what to look for while I coded away like a good little monkey. He tracked down the problem spots, I put in the cache control, and off we went.
We'll still be doing some maintenance and changes on the site. I don't know if we're out of the woods just yet with the performance tuning! But if I have to take one thing away from this it's never underestimate the power of a good profile.
-eli