Awesome custom metrics using Azure Application Insights

posted by Jeff | Sunday, February 23, 2020, 7:35 PM | comments: 0

You know that old management saying, that you can't improve what you don't measure? It's annoying because being measured feels personal and uncomfortable, but it's useful in technology because it's totally true. There is a fairly lucrative market around monitoring stuff in the cloud now, because as our systems become more distributed and less monolithic, we do less tracing at the code level and more digestion of interaction data between components.

When I launched the hosted forums, the big change was that I was finally exercising the two-level cache code that I wrote for POP Forums a long time ago. It was of course inspired by the way StackOverflow handles their caching. Basically, the web node checks to see if it has the object in its local memory, and if not, it goes to Redis. If it's not there either, then it goes to the database and puts it in Redis. Then, using Redis' pub/sub messaging, all of the web nodes get a copy and it holds on to it there with the same TTL. The default that I use is 90 seconds, but I don't know if that's "good" or not. I had no context or data to support that it worked at all, actually.

The first step for me was to plug in Azure Application Insights, because it's silly-easy to use. You add a NuGet package to your web app, a little Javascript to your pages, the app key to your configuration, and enjoy that sweet sweet stream of data in the Azure portal. Instantly I had this beautiful map of curvy lines going between databases and web apps and Azure functions and queues and... not Redis. OK, no problem, a little google-foo reveals that the SDK doesn't have the hooks to monitor Redis calls. So I wander through the Azure portal to my Redis instance, and look up the numbers there. What I find is that there are almost three times as many cache misses than hits, which implies that data isn't repeatedly fetched very often, or something else.

This is actually not super useful, because in theory, my two-level cache would not need to go all the way to Redis if the object is locally stored in the web node's memory. It's also not useful because I wanna see all of this in the pretty application maps!

OK, so with the underlying forum project being open source, I didn't want to bake in this instrumentation because I don't know what others are using. Not forcing them to use App Insights, I decided to wrap all of my cache calls in a simple start/stop interface, and wire in a default implementation that doesn't do anything. That way, a consumer can write their own and swap out the dependencies when they compose their DI container. For example, here's the code wrapping a write to the memory cache:

_cache.Set(key, value, options);
_cacheTelemetry.End(CacheTelemetryNames.SetMemory, key);

The _cache is an instance of IMemoryCache, and _cacheTelemetry is an instance of the interface I described above. One could argue that there are better ways to do this, and they're probably right. Ideally, since we're talking about something event based, having some kind of broader event based architecture would be neat. But that's how frameworks are born, and we're trying to be simple, so the default ICacheTelemetry is an implementation that does nothing.

In the hosted forum app, I use a fairly straightforward implementation. Let's take a look and then I'll explain how it works.

	public class WebCacheTelemetry : ICacheTelemetry
		private readonly TelemetryClient _telemetryClient;
		private Stopwatch _stopwatch;

		public WebCacheTelemetry(TelemetryClient telemetryClient)
			_telemetryClient = telemetryClient;

		public void Start()
			_stopwatch = new Stopwatch();

		public void End(string eventName, string key)
			var dependencyTelemetry = new DependencyTelemetry();
			dependencyTelemetry.Name = "CacheOp-" + eventName;
			dependencyTelemetry.Properties.Add("Key", key);
			dependencyTelemetry.Duration = new TimeSpan(_stopwatch.ElapsedTicks);
			dependencyTelemetry.Type = "CacheOp";

When you add Application Insights to your web project, you add services.AddApplicationInsightsTelemetry() to your Startup. Under the hood, this adds a shared instance of TelemetryClient to your DI container. This is what we use to pipe data back to Insights.

DANGER! A lot of the code samples in the SDK documentation show them creating new instances of the telemetry client with default configuration. This is naturally what I did first because I didn't read the documentation in any particular order. Something about this is bad, because my memory usage spiraled out of control until the app stopped responding. My theory is that it didn't have the right config, so it bottled up all of the events in memory and never flushed them, but I don't know for sure. Meh, using dependency injection is always better for testing anyway, so do that.

As shown earlier, the cache calls are wrapped in start and end calls, so all we're doing here is starting a timer, then stopping it and recording the cache event as a DependencyTelemetry object. For extra flavor, I'm recording the cache key as well, so if something is particularly slow, I can at least infer what the cached data was. For example, a key like "pointbuzz:PopForums.PostPages.17747" tells me that the tenant is "pointbuzz," and the rest says it's a bunch of posts from the topic with ID 17747.

I'm also giving the event a name that's prefixed with "CacheOp-" and not just the operation. Why? Because the lovely map view will group these together when they start with similar names. I learned this quite accidentally, because all of the database calls to tenant databases have a prefix on the database name. That worked out great because it groups the calls to tenant databases from the master database that describes the tenants.

Let's see some raw data! We can enter our Azure Insights in the portal, go to Logs, and query for the dependencies we said we would send:


There's a ton of context in every one of these entries, because they all roll up to the request data from the user. So yeah, I find a lot of weird cache misses from Russian bots that I assume are trying to spam forum topics that are 10 years old.

So that's cool, but remember that my original intent was understanding what the hit ratios were for the cache, for both in-memory and Redis. Well, we can write a query for that as well, and even pin it to a dashboard, if we so choose:

Behold! Useful data! This makes way more sense than my initial look at Redis data. It shows exactly what I would hope for, in fact: There are a ton of cache hits to the in-memory cache, and when those miss, they're persisted to memory and Redis. The bottom line is that the juicy data is right there in the memory most of the time. That's success! If you go back to the raw data above, you can see that those memory hits have a duration of zero, so less than a millisecond. That's obviously faster than crossing the wire to the database.

You can also get a good summary of what's going on, and track down slow hits, by going to the application map:

Right away, I wondered what the deal was with the slow gets from Redis. I'm still looking into it, but it looks like there's some overhead and warm up involved with the connection multiplexer in the StackExchange client. It's used so infrequently that there often isn't a connection to Redis established. That may partially explain why the SetRedis events are so short, because there had to be a connection to find there was nothing there before writing to it. It's all speculation at this point though, and generally pages are arriving in under 100ms.

Azure Application Insights is a very cool tool, and not limited to just the basics. It's important that your monitoring goes deeper, because the lesson I've learned time after time is that contextual monitoring is the thing that will save you. Nothing knows about the health of the system better than the system itself, so you should strive to have it tell you what's up (or down, as the case may be).

Technical validation at last!

posted by Jeff | Friday, February 21, 2020, 11:05 PM | comments: 0

Tuesday night, I migrated the PointBuzz forums to the hosted forum app, and the site that served as v1 more than 20 years ago came full circle as the first customer. Both of the sites had been running on a similar version for some time, but there's something different going on here that provides a mountain of technical validation that I didn't have before.

Without nerding too hard, I really wanted to prove that all of the stuff I wrote to scale the app worked. Not worked to scale, but worked at all. If you go back in history, the forums were always written to be this self-contained thing that would work on shared hosting (hello 1999!), so breaking it up into cloud-first pieces was a lot of work. But several big things worked out:

  • I moved search to ElasticSearch. In my case, I'm using the managed version from Elastic itself, running in Azure. I'm spending less than $20/month to start, and while it's not super fast at that spend, it gets real results that are useful! My home-grown solution wasn't terrible, but it wasn't great.
  • All of the background stuff that happens like sending email and indexing text for search, happens in serverless instances (Azure Functions). What's awesome about this is that it costs a few bucks for millions of executions.
  • It all works across multiple nodes! Since moving to the cloud, I always wanted to have more than one web head just for redundancy. There have only been a few times where I'm sure the node failed and a new one was spun up, but having two is nice. I have to keep the affinity on (sticky sessions) though in order to make SignalR work, but this doesn't seem to really impact performance.
  • The two-level caching works. I use StackOverflow's pattern here, sort of, replicating and invalidating the cache in local memory by way of the message bus in Redis. I don't have a great way to measure the hit/miss ratio in local memory, but I imagine it's pretty great. User data is what gets cached the most. I'm estimating that with PointBuzz alone, I'm avoiding the database about 50,000 times per day. That gets really important with more customers. (Sidebar: This failed a lot when I was prototyping a few years ago, as the old version of the StackExchange Redis client was prone to failure.)
  • Average response time went from about 30ms running on a Windows app service, to 5ms running on a Linux app service with about the same "hardware" and the same database. That's nuts! It's the same code! Maybe there's something I'm missing, but the only other thing that I can think of is that it's now running multi-tenant, which adds overhead, not reduces it.

I'm also trying out Azure Insights for monitoring. It's pretty cool, but the cost seems a little steep so far, I think averaging a buck a day for relatively low traffic. It also doesn't monitor the Redis service. Like any service, you can feed it custom events, so I might be doing that to instrument the caching both local and on Redis.

So now I just need customers. Selling is not my strong point, but I have some ideas to get the name out there and perhaps line up some "free" customers as promotion. I have a reasonably baked product, and that's very exciting.

Parenting status: exhausted

posted by Jeff | Thursday, February 20, 2020, 9:30 PM | comments: 0

I haven't written about our parenting adventures in a long time, in part because it's really hard to narrow it down to any particular category of action. Looking at it in the broadest of terms, we have this kid being a typical almost-10-year-old while rolling with aspects of ASD, ADHD and anxiety. It's a lot to process, and there are days where I just wish he could be a carefree kid that things come easy to. That certainly wasn't me as a child, so I'm not entirely sure why I would expect it to be true of him.

School this year is sometimes a struggle, though it's not as bad as it was last year. Without having a Type-A overachieving principal (I need to publish her emails about testing that I FOIA'd from the district some day) this year, Simon's homework load is lower, and they're not beating the kids up about standardized testing. He does get some homework, and it's a consistent struggle to get it done. This is where our team and the collective advice was not serving us. Sometimes for ADHD, you need to chunk-up work, which is a logical strategy. However, we were noticing that the ensuing freak-outs about getting it done could be easily diverted, the "squirrel!" moment (familiar if you're familiar with the movie Up). His developmental pediatrician, the one prescribing the drugs, immediately called bullshit on this as typical age-appropriate behavior, not ADHD. He's been working the system so he could do the things that he wants to do.

However, throwing this back to his therapist (which is not at all covered by insurance, we recently learned), she was a little taken aback by this. It puts us into a weird accommodation vs. accountability problem. Sometimes he does need to be granted accommodations, but not always. There's no magic formula for this, because it's completely contextual in the moment. It makes it harder for us and his teachers, who are bound by his IEP.

Then there's the video game problem. Let me first say that I can't in good conscience write off games as "bad." I loved them, and computers, as a kid, and this love was almost always treated as an annoyance or waste of time. I won't do that to Simon. I'm even more cautious now, because I'm finally seeing some level of creativity when he builds things in Minecraft. This was my concern about his Planet Coaster obsession, where he mostly downloaded other people's work, but never made anything. I believe creativity is one of the biggest contributing factors to success, and I want him to exercise these new muscles.

Now the issue is that it's one of the things he most cares about, and it can be at odds with doing homework. A funny thing happened when we said he couldn't play games after school until his homework was done, but he could have a necessary break before doing it. The homework got done in bigger chunks, a little faster. It isn't consistent yet, but it seems to be working. We'll see how that plays out. At the very least, it makes some of the learning challenges a little more apparent. For example, reading comprehension isn't really achieved by scanning for keywords, and he clearly does that. Knowing that, at least we can help.

We've asked the therapist to focus more on social skills and coping mechanisms for difficult social interaction. Simon's social struggles are fairly typical autism things, where he fundamentally doesn't get social contracts. This got him into potentially serious trouble at school when he joked about something inappropriate. You can imagine how that rolls for a kid who is under the impression that being funny is what makes people like you. Then throw in the random thing where kids say they aren't your friend for no particular reason, and you know how that goes. He desperately wants to have close friends, but at the same time can engage in really antisocial behavior. He is me, and it took me years (and a lot of therapy) to figure that out.

We're also in the midst of drug switching, again. The ADHD meds seem to be working OK, but we took him off of the anxiety drug because it clearly wasn't doing anything. The doctor replaced it with Abilify, which is used to treat a wide range of more serious conditions, but in his case, it's intended to even out his moods and help keep the anxiety in check. So far, it's hard to say if it's working, but it's definitely making him tired. Fatigue becomes yet another variable to consider, and then you're not sure if we're making progress. I suspect the ADHD drugs will stop working in the next 6 to 12 months too, just because none of them seem to help for a long duration. 

I hate this drug experimentation the most. There are no silver bullets, because everyone is different, but non-scientific me feels like I can't know who Simon really is. I need to get over that, because we know what no meds looks like, and it's a mind that's always racing.

All of that challenging stuff aside, we've certainly had some wins, too. Simon finally figured out how to ride a bike, thanks mostly to Diana's persistence over a few days. We bought him a bike two years ago, and he's finally riding it. In fact, he rides it to the bus stop now, locks it up, and rides it home after school. He is also (or was, before the drug switch) consistently getting himself out of bed in the morning and coming down for breakfast. Given our concern about accommodation that might have been coddling, you can imagine how good it is to see all of this personal responsibility.

While he doesn't generally fit in with many of the neighborhood kids, there are a couple that he seems to connect with at least some of the time. His interests are often about video games, so that's where the connection often is. It's difficult to teach him kindness when little is afforded to him, but we see it now and then. It's a work in progress.

We're a little exhausted. Parenting is not without joy, but I never imagined it would be this hard. Simon is a smart little boy, a little different at times, but I have to believe that someday it will work to his advantage. I find myself worrying less about his long-term outcomes lately, and more about him just having happy times as a child. When he's 30, I want him to believe that we did right by him, to the best of our ability.

Documentaries and Cheer

posted by Jeff | Saturday, February 15, 2020, 11:15 PM | comments: 0

Diana and I recently finished watching the 6-hour-ish documentary series Cheer on Netflix (or "docuseries," as apparently everyone involved in it is contractually obligated to say), and it was amazing. I thought while stream-surfing that maybe it was a reality show, but no, it was a bona fide documentary, and it was great. I went to a high school that actually took cheerleading seriously as a competitive sport, even if it wasn't in a local context, so I've always looked at it as a legitimate athletic endeavor. The documentary only reinforces that stance. I ended the series feeling like I wanted to adopt every single one of those kids and hug them and tell them that they're awesome. That's the mark of an effective documentary.

I've always had a thing for documentaries, and it somewhat relates back to the concept of "gonzo journalism" that Hunter Thomas brought about back in the day. The idea that you could observe something while influencing and participating in it, and still tell a story about it. For documentary film and television, this came about arguably with the start of MTV's The Real World. That's when they said, "This is the true story, of seven strangers, picked to live in a house, work together, and have their lives taped. Find out what happens, when people stop being polite, and start getting real." This was technically a documentary, but it was also a contrived and made up reality. Of course things get uncomfortable when you force a racist and a black activist to live together. There's nothing about it being "real" when you force it.

Documentary in its more pure form observes a situation that would have happened whether there were cameras there or not, and it's infinitely more satisfying, if somewhat voyeuristic, to see. Cheer checked all of those boxes. Our personal relation to it was that we stayed at a Daytona hotel next door to the band shell where the cheerleading championship took place. (For the record, Daytona, and those hotels, are a total shit hole.) We were there possibly around the time of the Cheer doc. I can't stand things like Survivor or The Bachelor, but Cheer was legit, because it wasn't contrived for TV.

I wish there was more of this on TV/streaming services. Unless you're a robot, documentaries are an instrument of empathy creation for your fellow humans. I think that's valuable. It's also the kind of thing that I aspire to. In high school I followed around a cheerleader during tryouts to show what that process was like, and that aired on cable access. In my first real job, I followed around high school freshman to show what it was like to start high school in an over-crowded school. I did a mini-doc after the fact to talk about the creation of a roller coaster. It's my thing even if I haven't pursued it on purpose.

The upside of the streaming-ization of TV is that there's more of this available than there used to be. That's exciting. I love documentaries.

The American Adventure

posted by Jeff | Saturday, February 15, 2020, 6:50 PM | comments: 0

If you've been to Epcot at Walt Disney World, you've probably seen The American Adventure, the audio-animatronic show that chronicles the history of the United States. The thing that I love about that attraction is that it's a surprisingly brutally honest review of our history. It doesn't gloss over slavery, the Civil War, the destruction of Native Americans or the wanton destruction of the environment. Despite all of the negatives, somehow the great experiment persists.

I always leave that attraction simultaneously proud, sad and hopeful. There's a montage at the end, to the original song "Golden Dream," where they show so much of our achievement, including shots of Space Shuttle astronauts, Dr. King, Kennedy's "Ask not" speech, famous Olympic athletes, Steve Jobs, Peter Jennings, etc. Our history is legitimately dark, but we've slowly been able to overcome some of it. It makes me want to be a part of the generation that helps us move above our worst.

I was a product of the tail end of the civil rights movement, specifically the part where desegregation was a forced issue in the schools. If environment is a deciding factor in how our views are shaped, I'm thankful for that experience. It's not that I don't see color because of it, it's that I see that color is not the determining factor in how people are valued. To not see color is to ignore the inequality that persists.

We were watching The Help today, and so at dinner, I had to explain to Simon what racism is since he wandered into the room while it was on. "Those people were stupid," he said, even though we're trying to discourage the use of that word when describing people. If it's really going to take another generation to shake racism out of our culture, hopefully I'm doing my part.

I desperately want that optimistic version of America in that Epcot attraction.

Making media

posted by Jeff | Friday, February 14, 2020, 12:40 AM | comments: 0

I had a great conversation today about content creation, and all of the various forms that takes on. Sometimes I forget that I had a previous career where writing, audio and video creation were my primary function. If I were to put a date on my flip to software developer, I'd probably put it in 1999, so I've been at that for 20 years. Since the flip, I have to remind myself that I wrote a book, a technical book, which is really hard and few people do that. Writing a book is like getting a tattoo. It'll always be there, and you'll always have a tattoo, er, be an author. I did over 200 podcast shows before podcasting was cool. I've squeezed out short documentary video pieces. I think my total spend on gear since 2006 has been nearly $15k (yikes, expensive hobby). I've written thousands of blog posts. I've also published a ton of stuff on that Internet, amateur design and what not.

All this to say that most of this content creation has come naturally, with joy and without any real planning, outside of the book. Recently, I've really enjoyed researching and writing the CoasterBuzz 20 pieces, seeing how much things have changed in that time, and how silly a kid I was. (Now I'm a silly middle-aged guy!) I've been watching documentaries and various Masterclasses from movie and writing people, and that has me energized. I actually opened Photoshop in the last two weeks, though I can't remember how to use it as effectively as I used to.

The act of creation has always been at the core of what I do, and I've come to realize that it's a core attribute of who I am. It's an intrinsic motivator. It's on a short list of things where I have to ask myself, "Is that I'm doing feeding this thing?" It applies to so many aspects of life, including parenting and professional endeavors like writing code and helping others develop their careers. Creation leads to outcomes.

There's something satisfying about realizing that I never truly left that previous career. I mean, I have a Discovery Channel credit for video!

Quietly taking your beating

posted by Jeff | Monday, February 10, 2020, 6:15 PM | comments: 0

I was watching something on TV talking about how social media is such a one-sided view of a person's life, made worse by the fact that some people use it as a means to build a persona that is impossibly unrealistic. That kind of bothers me. It's not really that different from real life. You rarely encountered water cooler chat before the Internet where people would describe their weekend as "terrible and here's why."

Keep in mind, I'm not talking about the Negative Nancy personality, which is intolerable in real life and on social media. That person who is always complaining about innocuous things that suck (coffee, lines for getting coffee, traffic, rain, etc.) is not fun to be around in any circumstances. But the person who is enduring hardship, or struggles with depression, we tend to have a cultural contract that expects us to repress these things. We need to figure out how to change that. Any time I've written about such things here, and it's rare despite the frequency, it has been met with empathy and "me too" reactions. Clearly the need to share is real.

Team Puzzoni has collectively been taking a beating since the year started. All three of us have had more than our share of difficulty, bad news, conflict and just general bullshittery. Some of the challenges are transient, some of them have long-term impacts, and most frustratingly, few have really been in our sphere of control. The last month has been exhausting.

Fortunately, we've got each other, and that's a big deal. It's still frustrating that it isn't cool to allow more bean spillage with the prospect of getting a random "you've got this" from a stranger. It feels like we live in a world that generally lacks empathy and people are all suspect of each other for a hundred stupid reasons.

So that's my release. We need a break, and I've said it out loud. Here's to better days going forward.

Inspiration from education

posted by Jeff | Sunday, February 9, 2020, 9:15 PM | comments: 0

I am a huge, huge fan of Masterclass. I know it isn't cheap, but it's worth every dime. The instructors are experts in a great many fields, not just creative, but also in business and science. The intent is not to become experts or achieve the level of the instructors, it's to have access to the wisdom and knowledge of people you wouldn't ordinarily get to meet. What you do with the information is entirely up to you.

This is the reason that we go to conferences in the software world. Being around the same people all of the time can make you dull. That's not a slight against them (except when it is), it's just what happens. Knowledge and wisdom comes from experience, and it's the one thing in life that you can't just naturally obtain independent of your environment. You have to "see the world."

We naturally gravitate to these learning experiences in our spare time, which if you think about it, reinforces that one of our greatest intrinsic motivators is in fact learning. Diana didn't know anything about quilting ten years ago, and today she wields a $10k long-arm machine like a boss. I'm that way with photography, and it could not have been a thing without learning from a great many resources.

It's my belief that as business leaders, we should take ongoing education more seriously. Admittedly, my line of work suffers from a serious gap between experience and need, so I might care about this more than practitioners of other fields. But in any business, it stands to reason that investing in your people in a non-trivial way is a thing where everyone wins. You demonstrate commitment to your people and they get better at their jobs. There's that old legend where a CEO asks the COO, "What if we teach them and they leave?" and the COO says, "What if we don't and they stay?"

Education, experience, learning... these are all undervalued in our culture even though they solve so many problems. We need to stop missing these opportunities. Critical thinking and observable evidence combined with an understanding of history can go a long way toward making us better humans.

Approaching multi-tenancy with cloud options

posted by Jeff | Tuesday, February 4, 2020, 10:45 PM | comments: 0

Building multi-tenancy into your app is an interesting (and dare I say fun) problem to solve, because there are a number of ways to approach it. And now that we don't have to spin up closets full of hardware in some basement, there are better options that we didn't have in the dark ages. I'll talk a little bit about the options that I've used, and how I solved the problem this time around with hosted POP Forums.

Establishing tenant context

To start, let's define it: Multi-tenancy is where an app has to accommodate a number of segregated customers, where the data doesn't overlap. This is different from just having multiple users, because while there are overlapping concerns between your users, they're all associated with a root customer. Think about the granddaddy of SaaS apps, Salesforce. When you're using it, you don't encounter the data of Company B down the street, only Company A, where you work. You're potentially using the same hardware instance, and definitely the same software, but you see different data. That's multi-tenancy.

Fundamentally, I believe your app needs what I call tenant context. Regardless of what part of the app is acting on something, whether it's a web-based action, API, some serverless thing (Azure Function or AWS Lambda), etc., it should know which tenant data it's working with. Setting the context is the first step of any action. At the web end of things, this is straight forward enough, because you could operate based on the domain name (, the path (, the query string (that's so 1999) or whatever your authentication and authorization mechanism is (auth tokens, for example). Some asynchronous thing, like a serverless component running in response to a queued message, may have tenant context baked into the message.

Partitioning the data

Once you have tenant context established, you have to decide how your data will be divided up. The strategies here are about as diverse as the persistence mechanisms. Many no-SQL databases already partition data by some kind of sub-key or identifier. The easiest way to isolate tenant data in a SQL database is to use a compound or composite key on every record, where you combine tenant ID and a typical identity field as your key. If you use Entity Framework, there are even a number of ways and libraries to add plumbing to all of your queries and commands to specify the tenant context, often by having your entities derive from a common base class that has a TenantID property. You could also divide up tenants by schema in SQL, or just provision a totally separate database entirely for each tenant. That's less scary than it sounds when you automate everything.

I've had to personally solve this problem several times over the years, and I never did it the same way twice. However, this allowed me to make some educated decisions when it came time to do it for POP Forums. Architecturally, the app consists of two web apps, one that serves the forums, and one that acts as the management interface for customers (billing, usage reporting and such), while a bunch of Azure Functions do a great many things in the background, triggered by timers and queue messages. The databases are SQL, using Azure SQL Elastic Pools.

Running code, in context

Let's walk through these parts. If we go to the source code, you'll find that the application is technically a monolith. The web app and Azure functions share code and they share the same database. (Moving compute around doesn't really break down your app into "microservices," because the parts still need each other and share state via a common database.) There is an interface called ITenantService, and the default implementation wired up in the dependency injection has no implemented SetTenant(tenantID) method. That's because if you're cloning this app and running it as is, there are no tenants.

Hosted POP Forums has two replacement implementations. For the web app, the SetTenant(tenantID) method puts the tenant ID into the HttpContext.Items collection. The GetTenant() method fetches it from the same place. (If you're wondering, there's a great new interface you can add to your container called IHttpContextAccessor which will get you to HttpContext, making all of this stuff easy to test.) Middleware placed early in the Startup sequence uses this implementation to set the context based on domain name. The web app also has a replacement for POP Forums' IConfig interface, which normally just reads values from a JSON config file. The replacement looks at the tenant context and builds up the right connection string, which is in turn used by all of the default SQL code. Obviously there is some error checking and redirecting for bad domain names and such, but that's (mostly) it for the web app.

The Azure Functions have an even simpler version of ITenantService. Since the functions are one-time use and thrown away, it gets the tenant ID from the queue message, stores it in the instance and reads it back as necessary. Relative to the open source code, there are not a ton of replacement implementations in the hosted version of the app. The biggest one writes error logs to a common instance of table storage in Azure instead of writing to the database.

So what about the timer based functions? The timers each queue messages in a fan-out pattern to run each task in tenant context. For example, one of the tasks is to clean up the session table, which feeds the "users online" list on the forum home page. It just nukes rows where the last updated time is too old. The timed function queues messages composed only of a tenant ID, one for each tenant. The listening function runs against that tenant.

Design rationale

One of the biggest requirements when I started the project was to modify as little as humanly possible from the open source project. I'm reasonably consistent about doing something with it on a weekly basis, and I don't want to change a consuming project beyond updated project references. So with that in mind, that's why I went with separate databases. Elastic pools make this economical and super easy, and when (if) I hit the ceiling for number of databases in a pool, it isn't hard to add some little bit of context about which "server" to look at for databases in the new pool. The downside, yes, is that any schema change has to be made in every database. Still, that's historically a small blast radius, and using something like DbUp, I could likely update hundreds of databases in a few minutes.

The domain name as key to defining the tenant is just obvious. People like to name things, and with additional automation, I can even provision custom domain names and certificates.

I've done some initial load testing in an unsophisticated way, and with two S1 app service nodes running on Linux and the old-school 100 DTU database pricing model, I can hit about 2,500 requests per second with sub-second response times. The unknown here is that my tests aren't hitting huge datasets, so the cache load (there is two-level caching going on with Redis and local memory) is not large. I'd like to design a better test and try P2v2 app services with three nodes, and flip the database to the vCore model. I think I'd need some impressive customers to really drag it down.

Overall, multi-tenancy was the easy part of the project. The more interesting parts were the recurring billing and provisioning. This week I'll be migrating one of our big sites on to the platform.

Foreign influence

posted by Jeff | Saturday, February 1, 2020, 6:20 PM | comments: 0

One of the things that I really liked about college was that, for a school in the middle of Nowhere, Ohio, there was a reasonably strong international student program. The residence life program, which I was a part of as a resident assistant for two years, also recruited heavily from that pool. After going to a high school that was essentially all white and suburban, it was my first exposure to a bigger world. It's also one of the reasons that I grew particularly interested in the people of India and their many cultures, because one of the other RA's in my building was Indian.

When I rolled into a software career a few years later, being around people not born here, either by way of immigration or work visa, became my default. Having not really sowed the seeds of travel desires until after I became a parent, I haven't had the opportunity to travel abroad as I'd like, so these folks have become my next best thing for now. I am fascinated by their stories, about what is different, and what is the same. I've made great professional friendships with people from India, Pakistan, Japan, Korea, China, Macedonia, Serbia, Croatia, Russia, Uzbekistan, Ukraine, Syria, Turkey, Germany, UK, Ireland, Australia, Mexico, Dominican Republic, Columbia, Venezuela, Brazil, Chile, Costa Rica, Argentina... and certainly others that I've forgotten. Oh, and Canada, eh?

As you might expect, this is one of the reasons that I love cruising as well. The cruise lines bring together people from all over the world who work their asses off for you. I enjoy the opportunity to talk to them about their families and the homes they go for months without seeing. My only wish is that they could spend more time being social and less working!

About three cruises ago, we had a server in Palo, one of the nicer upcharge restaurants on the Disney ships, who was unusually young, presumably 22 at the time. I say unusual, because it's typically the best servers who work up there, and excellence does come with experience, and therefore age. She was from Estonia, but most recently living outside of London. She was charming, smart, interesting, with the sort of kindness that you can't really fake. To that end, we requested to get her serving us the next two cruises, and were successful the last time. From there we followed her on social media. She mentioned that she would be passing through Orlando after her contract on the way to Columbia and needed a place to crash for a little less than 48 hours, so we offered. It seemed unlikely that she was an ax murderer.

We were pretty excited because it was a little like having an exchange student, only post-university age and free enough to travel the world. Her perspective on the world, and really life in general, seemed 10x more informed than my own when I was 23, and in general she was a lovely human being. Simon had a big sister for a day or so. It was actually the first time she ever spent time in the United States, in an American city, since to and from the cruise ship she never had the opportunity to move around. Hopefully we represented well, taking her out to the legendary Hamburger Mary's (between drag shows, unfortunately), and the timing was right to get cheap tickets for Aladdin at our beloved Dr. Phillips Center. Also, she was definitely not an ax murderer.

Our next international encounter will be with family friends from Norway this summer. That's very exciting. I might not be able to travel as much as I'd like to, but it's great when a few people can come to you. The world is a beautiful and diverse place, and I value the opportunity to meet the people in it.

CoasterBuzz is 20-years-old

posted by Jeff | Wednesday, January 29, 2020, 9:55 AM | comments: 0

There aren't many things that I can think of that I've done for 20 years. The next longest thing I can think of is my marriage, clocking in at almost 11 years. I've been a parent for less than 10. My longest running job was 3 years. But 20 years ago, I started CoasterBuzz, and it's still a thing.

Tomorrow is actually the official anniversary of the launch date in 2000, and to commemorate it, I've been writing some history down to share with people over the next few months. I've had a unique view of the industry and met a lot of interesting people. Some of the stories are self-indulgent, some are general interest, but whatever, it's my stories. The introduction that I wrote probably captures it all best:

What you're about to read is a one-sided account of 20 years of amusement park history and the web site that covered it. It's part memoir, part history, probably too inside-baseball and definitely more nostalgic than it should be.

I landed in this place quite by accident. Back in 1999, I was beginning to mourn the death of my broadcast career, after being pretty sure in college that I was going to own it. I was 25 when I started Guide To The Point (now known as PointBuzz), and 26 a year later when this site became a thing. There aren't any rules when you're 20-something, or at least, you ignore the rules, and the Internet definitely had no rules. I didn't know what I was doing, I just knew I enjoyed this new hobby of publishing without a printer, a video camera or a transmitter.

Over the years I got to do some really cool things. I was invited to countless roller coaster openings, created an online club, attracted unwanted attention from a ride manufacturer, acquired a professional friendship with a CEO, toured a factory where they made coaster track, and most importantly, built a network of friends that I never expected to have.

Online community in the roller coaster world has already peaked. Maybe it will be more prominent again someday, if niche communities can wrestle it all away from social media giants. But there was a time and place where this community, at CoasterBuzz, brought people together. A few years ago I was at a wedding reception at Walt Disney World. As we enjoyed the drinks and the fries and tots bar (best idea ever), someone made the observation that many of us knew each other primarily because of this site. Sure, I created it, but its existence was something bigger than me. It was the one thing we all had in common. For that, I will forever be grateful.

It's hard to predict the future, but it can be fun to look back. I hope you enjoy some of these stories. You probably won't be interested in all of them, but I hope at least a few of them prove interesting. Thank you for your support over the years!


Truthfully, I can't tell you how much longer I'll do it. I don't have any reason to stop, but I've definitely slowed down. There isn't as much news as there used to be, and huge ride installations like Millennium Force, while not exactly common, aren't setting new records or surprising the world. The industry doesn't have a lot of room to innovate, and there are only so many parks where you can install these great new rides.

Right now, the ad revenue from the site is funding my side project, cloud-hosted POP Forums, which may or may not be a bona fide business, but there are other companies doing it, so why not? My immediate focus for that is to get PointBuzz's forums moved there so we can decouple all of the content stuff from the forum and improve both independently.

So here's to two decades of nerding over roller coasters!

When you want to be a "technology company"

posted by Jeff | Tuesday, January 28, 2020, 8:35 PM | comments: 0

I've had a series of conversations in the last two weeks about the mentality with which companies approach the use and creation of technology. There's an attitude, more a spectrum of attitudes, that companies take toward tech, and it sets a tone and vastly affects outcomes. It ranges from viewing technology as a necessary evil to viewing the company as an entity that creates technology to arrive at some outcome. As you can guess, those companies that live closer to the latter scenario generally get it more right than those at the opposite end.

That doesn't mean that a pure technology player has its collective head in the right place though. There are stereotypical tech companies (that use words like "disruptor" and other nebulous terms) that think the tech itself is the most important thing, above the outcomes. It's a strange and arrogant thing to think that way, but there's an entire culture around it that originates largely from Silicon Valley. But I strongly believe that when a company views itself as a technology company that solves a problem, the outcomes are stronger.

My favorite anecdote about this was the group where I contracted for a short period of time at Progressive. I don't really know if this was a top-down intentional decision, but the leaders I had contact with were very deliberate in their language. They believed the company was a software company that sold insurance. As the first to really embrace online sales, away from the decades-old agency model, I believe that this was the wider intent. It made a difference in how they acted.

Another company that I worked at, or should I say endured for a year, was definitely in the "necessary evil" camp. It doesn't mean that they didn't spend money on technology. In fact, they spent an extraordinary amount, most of it on outside contractors and vendors, with a skeletal staff of B-players internally to try and glue it all together. Had the company viewed itself as a technology company pursuing strong outcomes in their space, I firmly believe that they would have made the creation and use of the technology a core competency, instead of a secondary concern. I'm reasonably certain that they would spend less money, too.

I think that even large companies can and do change when this attitude is taken from the top. Look at Disney... CEO Bob Iger's three high level goals were to invest in the best content (acquisitions of Marvel, Lucasfilm, Fox, etc.), be global and use technology to reach customers. That last part is the top-down direction I'm talking about, and it made Disney+ a thing. Ordinarily, I wouldn't trust a media company to figure that out. You can't phone it in. But there it is, and despite a rough first day or two, it's works pretty well. It took Netflix years to get to that scale. The media company made the decision to be a technology company that served media, instead of a media company that used technology. There is a fundamental difference in thinking there, even if your business itself is technically not the technology thing.

Sometimes I wonder if this is what keeps American business from innovating and winning (aside from the defense of incumbent industry). Think about Tesla's approach to making cars. They relentlessly focused on "the machines that build the machines," which is a different focus from building cars. From that learning process they opened up a plant in China in less than a year. Amazon started by selling books, but the machines that sold the books are now the business segment with the highest margins. The outcomes, the products, are important, but you can't approach getting there in spite of the technological need, you have to get there because of it.

Post-Christmas decoration withdrawal

posted by Jeff | Thursday, January 23, 2020, 7:30 PM | comments: 0

You may know this, but I'm married to a Christmas decoration junkie. The cheer that my darling wife Diana brings to our house every year is amazing and unmatched. We have no less than four Christmas trees in our house. In fact, we bought this particular floor plan in part because we knew the railings along the second floor would be awesome to wrap in lighted garland. Our electrical usage increases by 800 watts when the lights are all on. We tell the Google "Merry Christmas" and all of it comes on at once. (We tell it "Happy New Year" to turn it all off.)

If you were anything like me in college, you probably had Christmas lights up in your dorm room, and probably all year. Sometimes I wonder why we don't do that in our adult life. OK, so it would look junky, lights duct-taped to the wall, I get it, but it still seems weird that we don't do something like it as adults. I sort of do, because I have a bunch of Philips Hue lights in my living room, and go to funky colors when I want. But the duration of decoration, that seems too short. People are really critical about that, with stores selling stuff before Halloween, but when you don't have a real winter, it's different in Central Florida.

This year, our first decorations went up on November 11, and I enthusiastically endorse this. By Thanksgiving, almost everything was up. The last items didn't come down until this week, about half way into January. So yes, we have Christmas decorations up for about two months, or one-sixth of the year, and we're not ashamed.

Now that it's all down, it's kind of sad in here. Actually, that's just because we're enduring some cold and rainy weather, but it does feel like something is missing. Our general decor is not strong. We're not minimalists or anything, but we're far from the type of people who need to put something on every horizontal surface and hand something on every square inch of wall. And while we could hire an interior decorator, that would make the place feel like a model home because it's not "us" stuff.

The good news is that it'll all be back out in ten months. And there might be one tree still hidden in plain sight.

Revisiting the search for inspiration

posted by Jeff | Wednesday, January 22, 2020, 4:05 PM | comments: 0

I've been reading and viewing a lot of stuff about people and things that inspire people. I'm a big fan of catering to intrinsic motivators (while recognizing that people want to get paid what they're worth). A lot of things can inspire people, mostly the words and actions of other people. When you look at history in a wider context, you can see it everywhere. Dr. Martin Luther King's I Have A Dream speech is a big one on my mind given the holiday. Kennedy's declaration that the United States would go to the moon. Heck, 17-year-old Greta Thunberg's plea for action on climate change is a big deal.

I see inspiration in technology all of the time. In software, you can launch a business with very little capital because of the cheap cloud resources available now. We drive a pair of electric cars that cost half as much as they did five years ago, but have twice the range. More than half of our power, including that of the cars, comes from the sun. Most of the world's observable fact is recorded online and searchable from a super computer in my pocket.

But it's hard to focus on all of that positive when there's a whole lot of negative. In my lifetime, we've been taught to be afraid of nuclear weapons, then terrorism, then the economy, and with all of the associated scapegoats largely resigned to non-threat status, the culture has moved on to fear anyone not like you (if you're white, anyway). Especially in light of Dr. King's speech, it's disheartening to see that, in 50 years, we still haven't come far enough. It even feels like the world is making all of the same mistakes again.

There are a few lines in the song "The Stairs" by INXS that always kick me square in the goodies when I haven't heard the song in a long time. It's almost as if I hear them at precisely the right time:

Listen to by the walls
We share the same spaces
Repeated in the corridors
Performing the same movements

The nature of your tragedy
Is chained around your neck
Do you lead, or are you lead
Are you sure that you don't care?

Think about that... it calls out that we have more in common than not, and the ability to move forward is largely up to you. Inspiration is the product of what people do together. If we can't find the inspiration, perhaps we need to be the inspiration.

Using Azure DevOps pipelines for cloud-hosted POP Forums

posted by Jeff | Sunday, January 19, 2020, 11:35 AM | comments: 0

Last week I did my first production deployment of the cloud-hosted version of POP Forums. I'm really excited to offer a version that others can use without setting it up themselves (I'm really excited about the performance, too!). If you've followed me for any length of time, you know that I've been maintaining POP Forums as an open source project since around 2003. The new cloud-hosted version is about 95% the same code, with a few substitutions in the dependency injection container to facilitate multi-tenancy, especially in the asynchronous Azure Functions to make sure that they're acting on behalf of the right tenant. What I'm excited to share is that it's so easy to use the output of the open source project and get those bits into the commercial product.

Any time a commit or merge is made to the master branch of the open source project, a build is triggered in Azure DevOps. Beyond the actual building and running the unit test suite, there are several important steps:

  • dotnet pack: All of the assemblies are wrapped up in Nuget packages. POP Forums has a Razor View library that has all of the server-side UI bits and controllers, so as demonstrated in the sample project, you can run the thing strictly from packages.
  • nuget push: The packages are all pushed to MyGet.
  • gulp: The tasks in the gulpfile.js run to minify and package the scripts and css.
  • npm publish: The output of the front-end packaging is pushed to MyGet.
  • All of this is pushed out to a CI build site, where I can verify the changes. From commit/push to on-air takes about three and a half minutes.

This was all happening even before I decided to light up a hosted version of the app, because I was using it to power the forums on CoasterBuzz. Keep in mind that all of this was free to set up. GitHub is of course free for open source, and when you have a team of two, Azure DevOps is free as well. You can only run one build job at a time, but again, that's not a constraint.

As you may have guessed, the cloud-hosted POP Forums app references the packages on MyGet so it has the very latest bits ahead of the typical releases. The source for the commercial app is stored in a git repository in Azure DevOps, along side of its build pipelines and work items. Pushing to the master branch has a similar outcome, in that the new bits land on a development copy of the app, with configuration for fake payments and everything. It was here that I was able to test the recurring billing and maintenance tasks over the last few months.

The hosted app doesn't take a ton of additional schema to work, but there have been quite a few changes to it in the weeks leading up to release. For those, I used DbUp. I am seriously enamored with this library. It's always made me uncomfortable that most database versioning works on the premise of calculating the diff between old state and new state, and then generating SQL scripts to bridge the gap. If things get into a weird state in any one environment, your CI, QA or staging updates then are not necessarily exactly what will happen in your production deployment, and that could have unintended consequence. DbUp simply executes the exact same scripts at every update, in order, so your lower deployments are in fact a rehearsal of the production deployment.

The shorter story is that the DbUp project is a console app, so I ship that in the artifact, and a PowerShell call (on a Linux build agent!) runs it as part of the deployment pipeline.

Azure DevOps has an artifact-based deployment mechanism, so when I'm satisfied with the build that was deployed to the dev environment, I can specify manually specify that build be deployed to the production deployment. You can set up gated deployments as well, meaning that a user with the right permission has to click a button after a verified deployment to a lower environment, but I don't typically deploy to production as often.

Some other observations:

  • Did I mention that all of this automation is free? I doubt the cost to Microsoft is significant, and when someone in my career stage gets all of this for free and applies it to a day job environment with 40+ people in there, they certainly make it up with licenses.
  • Because the back-end code is all .Net Core, I can build on Linux build agents. When I switched from Windows-based agents, I shaved about a minute off of build times. (Not exactly related, but it should not be surprising that scaling app services up or out is also faster on Linux.)
  • The longest time of the build is package pulling (NuGet and npm). You can get around this if you host your own agents (not free), but since the free agents are ephemeral, they're going to pull every time. I'm surprised that they haven't found a way to cache for you, since it would be in their own interest to not tie up the free compute and incur the bandwidth cost of every pull.
  • The set up time for all of this is a fraction of what it used to be. I felt like I had to relearn all of this once a year, but it has been predictable and easy the last few years.

Overall, I'm really satisfied with the experience of using Azure DevOps for pipelines. Getting the value that you're creating in front of customers fast is easier (and cheaper) than ever. Go visit to see the hosted forums in action!

Nothing I write makes sense

posted by Jeff | Saturday, January 18, 2020, 12:20 PM | comments: 0

I've started at least four blog posts this week, and not one of them makes sense. Or maybe they make sense, but I feel like I'm unjustifiably complaining about things.

Diana and I are having "a week." Not with each other, thankfully, but in the general sense of life, this week hasn't been one of the stronger ones we've had. This has put my creative side in something of an incoherent place, so writing isn't working out as I would like. For CoasterBuzz's 20th anniversary, I'm writing some stories from the history I have, to start publishing at the end of the month, but they're going nowhere. If art comes from experiencing unpleasant things, I'm doing it wrong.

Next week has to be better.

It's not the death that worries me, it's the prioritization of life

posted by Jeff | Tuesday, January 14, 2020, 5:45 PM | comments: 0

Last weekend, a long-time blogger and educator specializing in the software stack that I've been around died at the age of 50. If that weren't weird enough, his last blog post appeared after he died. He had some great courses on Pluralsight. It's weird to think that any one blog post you make could be your last.

At the conference last week, I caught up with a former coworker that I had not seen in a decade. We had a lot of war stories to trade, mostly around the shared experience that strong business leaders are rare in software, from the executive level down to individual developers. This puts people in our career band, the managers between the makers and the C-suite, in a hard place as we serve two different audiences and act as the glue (or often lubricant) that makes it all work. It can be a thankless job, but we get enough out of it that we feel like it's worth it. Sometimes the big success is big when you've got it right on both sides, other times not so much. I suppose that's true for any job.

But being 40-somethings, approximately half way between wearing diapers and wearing diapers, we're acutely aware of the clock. I'd like to think that I've been at peace with my own demise for a long time. There isn't much point in fearing what you can't stop. What you can do is prioritize the time that you have to optimize for a meaningful and excellent life. That's the thing that stresses me out.

First you think in terms of relationships, and hopefully you start with your relationship with yourself. I probably spend too much time self-loathing for various reasons, but I'm at least self-aware of what's good or bad. I got lucky and scored a partner that is about as perfect as one gets. She's supportive, an amazing parent and my equal. This is the part I'm pretty good with. I'm spending that time right. I would lump parenting into the relationship bucket too, but once you start that, you can't really bail on it.

Work, whether it's the thing you do for money or the way you spend time to improve the world in some way, that all takes on a different meaning when you're optimizing for a good life. Sometimes you do crap work for a ton of money, or meaningful work for free, but it always feels like there's a priority aligned to it. For example, I once started a small team from scratch and built up a product that initially couldn't scale beyond a few users, and that felt good. Another time, I worked at a giant company where our team built a system that processed 100 million transactions per month. You bet that felt good. Another time I worked for a health insurance company trying to make their process better, and it was soul-sucking and terrible. I worked for an agency for like two months that wanted me to lie to customers. That didn't feel good either.

But most work probably lies somewhere in between. Our circumstances sometimes dictate that we have to do whatever it takes, which can certainly force the meaning curve. If we get lucky, we can often make our own story and have choices. But you never really know when you're taking your last shot, and you hope that the one in front of you is the right one.

I can't go to the Cleveland!

posted by Jeff | Friday, January 10, 2020, 5:40 PM | comments: 0

I'm back at CLE after the conference. CodeMash is so good, and I guess it's surprising just because you wouldn't expect a great conference to be in a winter-bound tourist area an hour away from Cleveland. Again this year, the quality of the people and content was amazing.

I had to drive through a light monsoon back to the airport, but it was easy compared to the worsening conditions two years ago as a mini-blizzard approached. But I was struck with the grayness and lack of color. As I was rolling up in the rental car shuttle, all I could think about is how badly I wanted to hibernate somewhere and take a nap. The memory was connected 100 different ways... college, high school, all of my jobs while living in Brunswick... all with a feeling of gloom and hibernation. I can't help but question: Are the unhappy memories a product of circumstances, or environment? I'm sure it's at least partly environment, because I had a lot of happy summer memories, but there are 36 winters worth of cold there too.

Objectively, Cleveland has had its shares of ups and downs. I remember the silly campaign, "New York is the Big Apple, but Cleveland is a plum." But it was a big deal when we scored the Rock-n-Roll Hall of Fame, and downtown had a (very) gradual renaissance that continues today. "White flight" gutted the parts of the city neighborhoods, and then later the reinvestment in those neighborhoods happened. The former Continental Airlines made the city a hub, but now the airport is half-empty, the "commuter" concourse closed. And while the Cavs finally had their championship moment, the Browns permanently suck.

What has always struck me though, was how loyal people were to the city. I was certainly a cheerleader for it back in the day. I still want it to succeed.

But I think that maybe I just outgrew it or something. I mean, I did leave Cleveland twice. After living in Seattle and Orlando, maybe I just felt like I'd been had, or completely self-unaware. Moving out of town really never occurred to me until the 2008 when the recession started to creep in and there was no work. The only other time I even considered moving a possibility was when I went to a conference in Portland, Oregon, way back when I was 25. Even that feeling quickly left me.

This is the first time I've been back in town since the Steel Vengeance opening at Cedar Point in 2018. That trip was suboptimal because, you guessed it, weather, and I was only there for 26 hours. The time before that was this conference the same year. I think we went for Cedar Point in 2016, and also in 2014 when we basically missed out because of the Great Water Main Break. It's like every time we come back, something borders on disaster.

I don't hate Cleveland, I just don't know what to do with it.

Hashtag conference life

posted by Jeff | Thursday, January 9, 2020, 5:30 PM | comments: 0

Wrapping up on the third day of conference activities at CodeMash in Sandusky, Ohio. Let me first say that it's a staggeringly great conference that goes deep, it's diverse in experience, people and technologies, and it's run by a non-profit. As a speaker, there's nothing to worry about except to show up. That's pretty great. This is my second time speaking at it, and I can't say enough good things about it. It started out many years ago as a small, regional, off-season thing, and now it's huge.

The first two days are long-form sessions, then the last two days are your typical hour-long deals. As a speaker, it's all included for me (they cover the room, too!), but I'm pretty sure the number of attendees doubles for the second half. This year I ended up kind of half-assing the long sessions, because I couldn't really clear all of my work obligations. I regret that, but it did give me time to finish up the deck for my talk, which was definitely not presentable when I landed at CLE. In fact, I really just got it to a good state an hour ago.

With tomorrow left, I'm mentally already pretty spent. There's a lot of content to take in, and then you fill up the in between times with conversations with people from all over. If you find time to talk to any of the vendors (which I've tried to pay more attention to this year, given my career stage), then you add that too. If that weren't enough, I'm trying to catch up with people from my old Ohio network, because so many are here. It's one of the few big driveable events around here.

The one positive about it being rural Ohio in winter is that it's not like there's a bunch of other things to distract you. Vegas conferences are kind of too much, the opposite extreme, where by the second day, you're kind of done with it all just because of what's outside the venue. Three days for these things is the sweet spot almost anywhere.

I'm satisfied, but ready to go home. The dry air I think gave me a sore throat, and Kalahari, where the conference is, has pretty bad beds. Today is also Diana's birthday, 50th no less, and no matter how many times she told me to go to this, I still hate missing the day. It'll be nice to see my little boy, too.

Making stuff in the down time

posted by Jeff | Sunday, January 5, 2020, 1:12 PM | comments: 0

The last two weeks were kind of like a forced vacation (I know, poor me). With the holidays falling on Wednesdays, I took the Mondays off as well, but in many ways I effectively took most of the two weeks off because few others were working. I did have some HR stuff, documentation and writing to do, but I probably put in at most 20 hours between the four "work" days. This caused me some anxiety, because as is often the case around the new year, you're setting goals and agendas for the coming year, and I couldn't really do that work. It's not really even that we have unlimited PTO, because companies that accrue it often force you to use it at the end of the year anyway.

I was able to focus some of that time into relaxation (and anxiety mitigation), including a nice little party on New Year's Eve that involved some nerds, sparklers, lasers and a fog machine. Outside of that, I poured some energy into my hosted forum project, and got a lot done. The software part is like 95% done, which is to say I can take money and make it work right now. That's exciting. The harder part is that there's a lot of polish and selling to do, which is frankly harder than writing code. I wrote a ton of documentation, and that's in good shape, but no real marketing piece just yet. I also have to write some boilerplate privacy policy and terms, which is even more boring. Once I get that all wrapped up, maybe next weekend, I'll advertise it and see what happens.

As I've said before, I'm looking for gravy income, mostly. If I can score 10 recurring customers before the end of the year, that would make me happy. Of course, I wouldn't shy away from 100 either, but that would create some interesting problems. The real intent though is just to prove that I can build something from scratch that's useful. I plan to move the PointBuzz forums into the system at some point, which will be great because it will allow us to keep the forums current and decouple them from the rest of the site, so they can change independently.

There's something deeply satisfying about sitting behind a computer that you built, writing code that does useful things (including collect money), automating the deployment and instrumentation of it all. Sure, you get this working in a team at work, but there's something different about doing it yourself, for fun. I guess some people build furniture or work on antique cars, but I make software.