At some point over the last six weeks and change, I launched POP Forums as a commercial, hosted product. By that I mean, it was something I could sell to others. I'm not suggesting that I've done any real sales effort, obvs, because I don't have any external customer yet. But there was a lot to learn from the effort.
The first bit was that, in instrumenting the shit out of it, I discovered some weaknesses early on. Maybe "weakness" is a strong word, because none of the things I found would have prevented a new, high scale customer from bringing it down. I had, over the course of the last year and a half, written all kinds of code to make it scale, but never really put the app into a real scale situation, where there were multiple nodes and an external service (ElasticSearch) providing search, etc. There were some quick lessons learned:
All of this still leaves me in awe that I can have this kind of redundancy and resiliency for so little money. How did we get along before "the cloud?" I've learned that more than 90% of my database calls are handled by in-memory cache (across multiple nodes, no less). The database DTU (the virtualized unit of CPU for the price structure I'm using) averages less than 4%, while the CPU for the web nodes stays well under 10% at the current, starter pricing levels. Page render times stay below 40ms, and that's accounting for serving images out of the database. I'm still confident that I could hit 2,500 request per second without scaling up (though response times would be better if I did).
The "big" test came when I migrated the PointBuzz forums over to the platform, and wow did it shine. It's not a huge load, but 30 to 50 requests per second isn't unusual for that site, and it doesn't cause any deviation from the above numbers. I'm so happy to split the forums from the rest of the site, so they aren't dependent on each other. The search doesn't get a lot of use there, but it's so nice to have real, useful search! The migration also forced me to add a feature to insert Google AdSense ad slots into the page, because their "auto ad" thing wasn't really inserting anything. I added code to insert a banner at the top, and a sidebar ad, and wouldn't you know it, it's generating CPM's of almost $2. Haven't seen money like that in a long time!
At this point, I just need some real customers to prove this is all for real. Honestly, it all works better than I expected.