So much sucky software, plus update on my new server

I find it amazing that there is so much crappy software out there. In my entire career, I've only had one job where most of what was built before I started was solid. In my current job, I suppose it could be argued that what we have isn't the worst since it does generally work (no, seriously, working isn't as high a bar to shoot for as you'd think), but most of what was done by the team prior to ours is far from optimal.

For example, we found out after changing some of our caching infrastructure that the app was hitting the database to lookup an anonymous user. A lot. If you're saying, "Why would you look up someone who is anonymous," yeah, you get what I'm saying. It's bad news when you're pushing 24 CPU cores to the brink. What frustrates me is not the lack of optimization, but rather the complete disregard for thinking, "How can I get away with doing as little as possible to serve this Web page?"

I recall a consulting gig I had in 2005, where we were trying to save an app where a couple dozen people were doing data entry from Windows client apps. The developer they worked with did the classic wrong thing: Get vague requirements from the client, disappear for a few months, and return with an app that failed. They had a database that was choking because it was poorly designed. Given the number of users, it should have barely got warm.

As much as I like to bitch about this kind of thing, I prefer to think through the solutions. Most issues like this are the result of developing for a big bang moment. That's a ton of planning (read: guessing), and putting something out there that simply isn't going to work the way you hope despite the planning. That's why I'm such a fan of agile development, iterating quickly and keeping everyone involved. A bug isn't a big deal when you only have to live with it for a week or two!

And speaking of performance, I'm happy to report the new server is so much over-kill that it's funny. The graph below is CPU usage when it's busy. No joke. The old server generally ran about 5 to 10% at peak times, with little spikes here and there. This one is less than 1%. Disk usage almost doesn't exist, probably because it has four times the RAM. Is it worth the extra $64 a month? Hard to say. Most pages on all sites render in under one tenth of a second, and Google's Webmaster tools say the average download time is around 110 ms. That's pretty awesome. We had one of our biggest traffic days in years on Monday (for a tragic reason, unfortunately), and the server still never got even remotely stirred. The benefit to me is less about the better hardware and more about the efficiency and speed of IIS7. Happy to finally be up to date.

On that note, I wanna go work on my own stuff. I haven't done that in about a week.