The long history of "devops" and my hobby development projects

I submitted a speaking abstract for an event last week about how you can use all kinds of free stuff to do continuous integration for an open source project. Also just yesterday, at work we saw Microsoft light up some great integration between GitHub and Azure DevOps (formerly called VSTS, among other things). We're at a point where we're starting to take all of this automation for granted, but I thought it would be amusing to look back at where my own efforts have taken me over the years in the context of my hobby projects. Buckle up... it's a long story. 

If I go back to the launch of Guide to The Point (now called PointBuzz) in 1998, the site was a bunch of static HTML pages. I used an app called Microsoft FrontPage to work with all of the files, but was still mostly manipulating the HTML directly. I used some shared hosting account that was $20 per month. Here's where it gets more ridiculous: FrontPage was actually using extensions on the server so I could manipulate the files directly (over a modem and land line, of course). Every now and then I would use FTP to make a copy of them locally, but I was editing on the live server. There was a Perl-based bulletin board app called Ultimate Bulletin Board (UBB) that ran in its own folder.

By the end of 1999, I had written my own forum app in the old scripting platform called ASP, using the VBscript language. It actually had SQL injection vulnerabilities here and there because I did some dumb things, though none were exploited because bots and script kiddies weren't a thing then. I was still editing files directly on the server with FrontPage.

In 2000, I launched CoasterBuzz  and spent much of the year improving the forum, but my workflow didn't really change. My traffic on the sites was getting ridiculous, so I rented a dedicated server from a company called CrystalTech for a base $350 per month. They were getting annoyed with me because I was using a ton of bandwidth, on the order of 50 gigabytes per month, so I was starting to spend $600+ each month. Advertising was fortunately covering it, but ouch. I was starting to sell the forum app, too, which meant spending a few hundred dollars on a secure certificate for the site, plus fees for a merchant account. The important change during this time was that I installed Visual Source Safe, a very crappy source control system that Microsoft had at the time, on that server. I was still editing files on the server though.

In 2001, CrystalTech was giving me crap about the bandwidth, so I looked into getting a T-1 to my house. Sprint hooked me up, and it was about $1,000 per month, plus the cost of a server I built, SQL license and a Cisco router. Things went to shit that year, with the dotcom bust and 9/11, so I lost my job and my key advertising provider, Doubleclick (which would eventually be bought by Google for the ad serving tech), dropped me. That's the year I started CoasterBuzz Club, which saved my ass.

In 2002 I updated the forum and sites to run on ASP.NET Webforms. I finally started developing stuff locally, and using some combination of Visual Studio and FTP to deploy the sites manually to the server under my desk. And when I screwed up, I had everything in source control. After two years with the T-1, dedicated servers came down in price, and I was spending around $300 per month instead of a grand. By the end of that year, I started giving away the forum for free, because no one would buy it with other free options available.

It was some time in 2004 or 2005 that I switched from Visual Source Safe to Subversion as my source control, again, installed on my server. Beyond that, really nothing much had changed for years. Then, in 2010, while working at Microsoft in the group that also maintained CodePlex (I worked on MSDN/TechNet forums), I moved the source control for the forums to CodePlex so it could be a true open source project. I think I also switched to Mercurial at that time, which was the hotness for awhile and comparable to Git. I recall we used it on MSDN, and I think CodePlex itself was using Mercurial for source control. I didn't actually do a release for the forums on CodePlex until 2011, when I finally finished a conversion to the MVC framework on top of ASP.NET, with unit tests and everything!

Active development on CodePlex ended I believe around 2014, much to the dismay of me and some of my former teammates in Redmond. They didn't announce its closure until a few years later, but knowing of its imminent death, I moved POP Forums to GitHub around that time, which also means I obviously switched entirely to Git for source control.

Also in 2011, Microsoft started testing a cloud version of Team Foundation Server (TFS), and I signed up to be an internal beta tester. TFS had its own source control I think, but they were also going to support Git, which was exciting because CodePlex was doing that too. It was pretty rough, which is to say it barely worked. Even though I left the company late that year, I was able to stay in the beta program. In 2012 they started a public preview, and at that point I moved some of the Subversion repositories I had from my rented server (which had changed companies three times) to Git on what would officially open as Visual Studio Online in 2013. They would rebrand it as Visual Studio Team Services then this year as Azure DevOps. Somewhere in this window, probably closer to 2011, I started using Microsoft's "web deploy," which is a dumb name, to uh, deploy my sites via the web. It's tech that already sits in Visual Studio and most any Windows-based build system, so it's super convenient and very fast.

In 2014, the cost structure for cloud resources, specifically platform (PaaS) offerings and not infrastructure (IaaS), finally beat the curve and allowed me to migrate everything to Azure. That meant no more feeding servers with patches, maintaining backups, cleaning up logs or any of that nonsense. I also lit up a virtual machine there of the cheapest kind, and put my old remaining Subversion repositories there. That VM still has some of the old file structure from the dedicated servers, even though I keep it "off" so only the storage costs me these days.

Starting around 2005-ish, every job I had involved setting up some kind of continuous integration pipeline of building and deploying code. I'm not sure how it ended up being me in these jobs, but the tools were pretty terrible, and every year I had to relearn how to do it. I think at Microsoft we were using Team City and some hacky Powershell scripts to run msbuild or something, and it wasn't until I got to SeaWorld Entertainment where the tooling was intuitive enough to feel obvious. The basic tooling shell and gated nature of those pipelines is still there, but now we've thrown in constantly evolving things to package up front-end code and a lot of scripting to push out to any number of compute flavors.

Despite this work history, it wasn't until 2016 that I fully automated a CI build and release pipeline for POP Forums, the open source project I still maintain (though not very actively, so there's some ancient crappy code in there). It took a few years for the next iteration of .Net, Core, to stabilize, so I finally felt like it was worth it to deploy the latest bits somewhere that people could mess with. This year I went even further by publishing NuGet packages of the code as part of the build, and I use those packages to feed CoasterBuzz when I'm developing it (PointBuzz still uses the "old" version of ASP.NET MVC).

So my workflow today, and this goes a long way from where I started, pushing static HTML to a shared server, is this:

• Commit and push code to GitHub for POP Forums.
• Azure DevOps is triggered by GitHub to build the project with all dependencies and run all of the unit tests.
• If the build is successful, it builds packages that are published to MyGet (I still need to add a step for an npm package, but I'll get to that when there's something interesting to publish).
• The build is deployed via web deploy to an Azure App Service where the world can mess with it. All configuration is stored in the definition of the App Service, entered via the Azure portal.
• CoasterBuzz is cloned from a private Git repository in Azure DevOps, and I work on it entirely locally.
• The CB solution references the packages on MyGet, so I update them as needed. Even the views from the forum are in a package.
• When I build it locally and it's ready with passing unit tests, I use web deploy from within Visual Studio to deploy CB to the staging slot in the app service, which uses the same configuration and same database (because changes are so rare).
• I smoke test and validate the changes in the staging slot, and if everything is good, I click a button in the Azure portal that swaps the staging and production instances, and my changes are on the air.

The costs are fairly inexpensive as well. Here is the rough break down for all of the sites:

• A single S1 service plan (1 CPU core, 1.75 gigs of RAM) hosts all of the sites at a cost of about $74/month. It averages about 80% memory utilization and CPU peaks around 5% during the day at most. I'm super proud at how fast it is, with page renders under 100ms.
• An S0 SQL database for each site, at $15/month each, and another $5 each for a B database for the forum test site, my blog and PointBuzz Live (our live blog site). That's $45 total for all of the databases. CB and PB get 10 "DTU's," which is their abstraction around "transaction units," or how much you can push stuff in and out of the database. Typical load is around 20% during the day, but it tends to spike to 80% at times, in short bursts. The bursts come from the inefficient self-rolled search indexing I build into the forums. I would use Azure Search for this, but their pricing is terrible, based on storage instead of computing power.
• Bandwidth is somewhere around 120 gigs most months, and costs about $10. Remember, that first dedicated server I had wanted $200 for less than half of that bandwidth back in 2001.
• There are all kinds of things that come "free" with all of this, like the fact that there are always about three copies of the databases, and if the "server" ever dies for the sites, it transparently spins up another one. Right now this system barely breaks a sweat at 5 to 10 requests per second (that's not counting fetching static things like images or scripts), but I can scale it up almost instantly to the point it could handle 250 requests per second easily.

It's a crazy different world from 1998. It's still nothing compared to some of the things I've worked at in various jobs, like the MSDN reputation/profile system, which even then handled 100 million transactions per month. Still, all of this amazing technology to use, and much of it is free.