Archive: February, 2020

You know that old management saying, that you can't improve what you don't measure? It's annoying because being measured feels personal and uncomfortable, but it's useful in technology because it's totally true. There is a fairly lucrative market around monitoring stuff in the cloud now, because as our systems become more distributed and less monolithic, we do less tracing at the code level and more digestion of interaction data between components.

When I launched the hosted forums, the big change was that I was finally exercising the two-level cache code that I wrote for POP Forums a long time ago. It was of course inspired by the way StackOverflow handles their caching. Basically, the web node checks to see if it has the object in its local memory, and if not, it goes to Redis. If it's not there either, then it goes to the database and puts it in Redis. Then, using Redis' pub/sub messaging, all of the web nodes get a copy and it holds on to it there with the same TTL. The default that I use is 90 seconds, but I don't know if that's "good" or not. I had no context or data to support that it worked at all, actually.

The first step for me was to plug in Azure Application Insights, because it's silly-easy to use. You add a NuGet package to your web app, a little Javascript to your pages, the app key to your configuration, and enjoy that sweet sweet stream of data in the Azure portal. Instantly I had this beautiful map of curvy lines going between databases and web apps and Azure functions and queues and... not Redis. OK, no problem, a little google-foo reveals that the SDK doesn't have the hooks to monitor Redis calls. So I wander through the Azure portal to my Redis instance, and look up the numbers there. What I find is that there are almost three times as many cache misses than hits, which implies that data isn't repeatedly fetched very often, or something else.

This is actually not super useful, because in theory, my two-level cache would not need to go all the way to Redis if the object is locally stored in the web node's memory. It's also not useful because I wanna see all of this in the pretty application maps!

OK, so with the underlying forum project being open source, I didn't want to bake in this instrumentation because I don't know what others are using. Not forcing them to use App Insights, I decided to wrap all of my cache calls in a simple start/stop interface, and wire in a default implementation that doesn't do anything. That way, a consumer can write their own and swap out the dependencies when they compose their DI container. For example, here's the code wrapping a write to the memory cache:

_cacheTelemetry.Start();
_cache.Set(key, value, options);
_cacheTelemetry.End(CacheTelemetryNames.SetMemory, key);

The _cache is an instance of IMemoryCache, and _cacheTelemetry is an instance of the interface I described above. One could argue that there are better ways to do this, and they're probably right. Ideally, since we're talking about something event based, having some kind of broader event based architecture would be neat. But that's how frameworks are born, and we're trying to be simple, so the default ICacheTelemetry is an implementation that does nothing.

In the hosted forum app, I use a fairly straightforward implementation. Let's take a look and then I'll explain how it works.

	public class WebCacheTelemetry : ICacheTelemetry
	{
		private readonly TelemetryClient _telemetryClient;
		private Stopwatch _stopwatch;

		public WebCacheTelemetry(TelemetryClient telemetryClient)
		{
			_telemetryClient = telemetryClient;
		}

		public void Start()
		{
			_stopwatch = new Stopwatch();
			_stopwatch.Start();
		}

		public void End(string eventName, string key)
		{
			_stopwatch.Stop();
			var dependencyTelemetry = new DependencyTelemetry();
			dependencyTelemetry.Name = "CacheOp-" + eventName;
			dependencyTelemetry.Properties.Add("Key", key);
			dependencyTelemetry.Duration = new TimeSpan(_stopwatch.ElapsedTicks);
			dependencyTelemetry.Type = "CacheOp";
			_telemetryClient.TrackDependency(dependencyTelemetry);
		}
	}

When you add Application Insights to your web project, you add services.AddApplicationInsightsTelemetry() to your Startup. Under the hood, this adds a shared instance of TelemetryClient to your DI container. This is what we use to pipe data back to Insights.

DANGER! A lot of the code samples in the SDK documentation show them creating new instances of the telemetry client with default configuration. This is naturally what I did first because I didn't read the documentation in any particular order. Something about this is bad, because my memory usage spiraled out of control until the app stopped responding. My theory is that it didn't have the right config, so it bottled up all of the events in memory and never flushed them, but I don't know for sure. Meh, using dependency injection is always better for testing anyway, so do that.

As shown earlier, the cache calls are wrapped in start and end calls, so all we're doing here is starting a timer, then stopping it and recording the cache event as a DependencyTelemetry object. For extra flavor, I'm recording the cache key as well, so if something is particularly slow, I can at least infer what the cached data was. For example, a key like "pointbuzz:PopForums.PostPages.17747" tells me that the tenant is "pointbuzz," and the rest says it's a bunch of posts from the topic with ID 17747.

I'm also giving the event a name that's prefixed with "CacheOp-" and not just the operation. Why? Because the lovely map view will group these together when they start with similar names. I learned this quite accidentally, because all of the database calls to tenant databases have a prefix on the database name. That worked out great because it groups the calls to tenant databases from the master database that describes the tenants.

Let's see some raw data! We can enter our Azure Insights in the portal, go to Logs, and query for the dependencies we said we would send:

There's a ton of context in every one of these entries, because they all roll up to the request data from the user. So yeah, I find a lot of weird cache misses from Russian bots that I assume are trying to spam forum topics that are 10 years old.

So that's cool, but remember that my original intent was understanding what the hit ratios were for the cache, for both in-memory and Redis. Well, we can write a query for that as well, and even pin it to a dashboard, if we so choose:

Behold! Useful data! This makes way more sense than my initial look at Redis data. It shows exactly what I would hope for, in fact: There are a ton of cache hits to the in-memory cache, and when those miss, they're persisted to memory and Redis. The bottom line is that the juicy data is right there in the memory most of the time. That's success! If you go back to the raw data above, you can see that those memory hits have a duration of zero, so less than a millisecond. That's obviously faster than crossing the wire to the database.

You can also get a good summary of what's going on, and track down slow hits, by going to the application map:

Right away, I wondered what the deal was with the slow gets from Redis. I'm still looking into it, but it looks like there's some overhead and warm up involved with the connection multiplexer in the StackExchange client. It's used so infrequently that there often isn't a connection to Redis established. That may partially explain why the SetRedis events are so short, because there had to be a connection to find there was nothing there before writing to it. It's all speculation at this point though, and generally pages are arriving in under 100ms.

Azure Application Insights is a very cool tool, and not limited to just the basics. It's important that your monitoring goes deeper, because the lesson I've learned time after time is that contextual monitoring is the thing that will save you. Nothing knows about the health of the system better than the system itself, so you should strive to have it tell you what's up (or down, as the case may be).

Tuesday night, I migrated the PointBuzz forums to the hosted forum app, and the site that served as v1 more than 20 years ago came full circle as the first customer. Both of the sites had been running on a similar version for some time, but there's something different going on here that provides a mountain of technical validation that I didn't have before.

Without nerding too hard, I really wanted to prove that all of the stuff I wrote to scale the app worked. Not worked to scale, but worked at all. If you go back in history, the forums were always written to be this self-contained thing that would work on shared hosting (hello 1999!), so breaking it up into cloud-first pieces was a lot of work. But several big things worked out:

• I moved search to ElasticSearch. In my case, I'm using the managed version from Elastic itself, running in Azure. I'm spending less than $20/month to start, and while it's not super fast at that spend, it gets real results that are useful! My home-grown solution wasn't terrible, but it wasn't great.
• All of the background stuff that happens like sending email and indexing text for search, happens in serverless instances (Azure Functions). What's awesome about this is that it costs a few bucks for millions of executions.
• It all works across multiple nodes! Since moving to the cloud, I always wanted to have more than one web head just for redundancy. There have only been a few times where I'm sure the node failed and a new one was spun up, but having two is nice. I have to keep the affinity on (sticky sessions) though in order to make SignalR work, but this doesn't seem to really impact performance.
• The two-level caching works. I use StackOverflow's pattern here, sort of, replicating and invalidating the cache in local memory by way of the message bus in Redis. I don't have a great way to measure the hit/miss ratio in local memory, but I imagine it's pretty great. User data is what gets cached the most. I'm estimating that with PointBuzz alone, I'm avoiding the database about 50,000 times per day. That gets really important with more customers. (Sidebar: This failed a lot when I was prototyping a few years ago, as the old version of the StackExchange Redis client was prone to failure.)
• Average response time went from about 30ms running on a Windows app service, to 5ms running on a Linux app service with about the same "hardware" and the same database. That's nuts! It's the same code! Maybe there's something I'm missing, but the only other thing that I can think of is that it's now running multi-tenant, which adds overhead, not reduces it.

I'm also trying out Azure Insights for monitoring. It's pretty cool, but the cost seems a little steep so far, I think averaging a buck a day for relatively low traffic. It also doesn't monitor the Redis service. Like any service, you can feed it custom events, so I might be doing that to instrument the caching both local and on Redis.

So now I just need customers. Selling is not my strong point, but I have some ideas to get the name out there and perhaps line up some "free" customers as promotion. I have a reasonably baked product, and that's very exciting.

I haven't written about our parenting adventures in a long time, in part because it's really hard to narrow it down to any particular category of action. Looking at it in the broadest of terms, we have this kid being a typical almost-10-year-old while rolling with aspects of ASD, ADHD and anxiety. It's a lot to process, and there are days where I just wish he could be a carefree kid that things come easy to. That certainly wasn't me as a child, so I'm not entirely sure why I would expect it to be true of him.

School this year is sometimes a struggle, though it's not as bad as it was last year. Without having a Type-A overachieving principal (I need to publish her emails about testing that I FOIA'd from the district some day) this year, Simon's homework load is lower, and they're not beating the kids up about standardized testing. He does get some homework, and it's a consistent struggle to get it done. This is where our team and the collective advice was not serving us. Sometimes for ADHD, you need to chunk-up work, which is a logical strategy. However, we were noticing that the ensuing freak-outs about getting it done could be easily diverted, the "squirrel!" moment (familiar if you're familiar with the movie Up). His developmental pediatrician, the one prescribing the drugs, immediately called bullshit on this as typical age-appropriate behavior, not ADHD. He's been working the system so he could do the things that he wants to do.

However, throwing this back to his therapist (which is not at all covered by insurance, we recently learned), she was a little taken aback by this. It puts us into a weird accommodation vs. accountability problem. Sometimes he does need to be granted accommodations, but not always. There's no magic formula for this, because it's completely contextual in the moment. It makes it harder for us and his teachers, who are bound by his IEP.

Then there's the video game problem. Let me first say that I can't in good conscience write off games as "bad." I loved them, and computers, as a kid, and this love was almost always treated as an annoyance or waste of time. I won't do that to Simon. I'm even more cautious now, because I'm finally seeing some level of creativity when he builds things in Minecraft. This was my concern about his Planet Coaster obsession, where he mostly downloaded other people's work, but never made anything. I believe creativity is one of the biggest contributing factors to success, and I want him to exercise these new muscles.

Now the issue is that it's one of the things he most cares about, and it can be at odds with doing homework. A funny thing happened when we said he couldn't play games after school until his homework was done, but he could have a necessary break before doing it. The homework got done in bigger chunks, a little faster. It isn't consistent yet, but it seems to be working. We'll see how that plays out. At the very least, it makes some of the learning challenges a little more apparent. For example, reading comprehension isn't really achieved by scanning for keywords, and he clearly does that. Knowing that, at least we can help.

We've asked the therapist to focus more on social skills and coping mechanisms for difficult social interaction. Simon's social struggles are fairly typical autism things, where he fundamentally doesn't get social contracts. This got him into potentially serious trouble at school when he joked about something inappropriate. You can imagine how that rolls for a kid who is under the impression that being funny is what makes people like you. Then throw in the random thing where kids say they aren't your friend for no particular reason, and you know how that goes. He desperately wants to have close friends, but at the same time can engage in really antisocial behavior. He is me, and it took me years (and a lot of therapy) to figure that out.

We're also in the midst of drug switching, again. The ADHD meds seem to be working OK, but we took him off of the anxiety drug because it clearly wasn't doing anything. The doctor replaced it with Abilify, which is used to treat a wide range of more serious conditions, but in his case, it's intended to even out his moods and help keep the anxiety in check. So far, it's hard to say if it's working, but it's definitely making him tired. Fatigue becomes yet another variable to consider, and then you're not sure if we're making progress. I suspect the ADHD drugs will stop working in the next 6 to 12 months too, just because none of them seem to help for a long duration. 

I hate this drug experimentation the most. There are no silver bullets, because everyone is different, but non-scientific me feels like I can't know who Simon really is. I need to get over that, because we know what no meds looks like, and it's a mind that's always racing.

All of that challenging stuff aside, we've certainly had some wins, too. Simon finally figured out how to ride a bike, thanks mostly to Diana's persistence over a few days. We bought him a bike two years ago, and he's finally riding it. In fact, he rides it to the bus stop now, locks it up, and rides it home after school. He is also (or was, before the drug switch) consistently getting himself out of bed in the morning and coming down for breakfast. Given our concern about accommodation that might have been coddling, you can imagine how good it is to see all of this personal responsibility.

While he doesn't generally fit in with many of the neighborhood kids, there are a couple that he seems to connect with at least some of the time. His interests are often about video games, so that's where the connection often is. It's difficult to teach him kindness when little is afforded to him, but we see it now and then. It's a work in progress.

We're a little exhausted. Parenting is not without joy, but I never imagined it would be this hard. Simon is a smart little boy, a little different at times, but I have to believe that someday it will work to his advantage. I find myself worrying less about his long-term outcomes lately, and more about him just having happy times as a child. When he's 30, I want him to believe that we did right by him, to the best of our ability.

Diana and I recently finished watching the 6-hour-ish documentary series Cheer on Netflix (or "docuseries," as apparently everyone involved in it is contractually obligated to say), and it was amazing. I thought while stream-surfing that maybe it was a reality show, but no, it was a bona fide documentary, and it was great. I went to a high school that actually took cheerleading seriously as a competitive sport, even if it wasn't in a local context, so I've always looked at it as a legitimate athletic endeavor. The documentary only reinforces that stance. I ended the series feeling like I wanted to adopt every single one of those kids and hug them and tell them that they're awesome. That's the mark of an effective documentary.

I've always had a thing for documentaries, and it somewhat relates back to the concept of "gonzo journalism" that Hunter Thomas brought about back in the day. The idea that you could observe something while influencing and participating in it, and still tell a story about it. For documentary film and television, this came about arguably with the start of MTV's The Real World. That's when they said, "This is the true story, of seven strangers, picked to live in a house, work together, and have their lives taped. Find out what happens, when people stop being polite, and start getting real." This was technically a documentary, but it was also a contrived and made up reality. Of course things get uncomfortable when you force a racist and a black activist to live together. There's nothing about it being "real" when you force it.

Documentary in its more pure form observes a situation that would have happened whether there were cameras there or not, and it's infinitely more satisfying, if somewhat voyeuristic, to see. Cheer checked all of those boxes. Our personal relation to it was that we stayed at a Daytona hotel next door to the band shell where the cheerleading championship took place. (For the record, Daytona, and those hotels, are a total shit hole.) We were there possibly around the time of the Cheer doc. I can't stand things like Survivor or The Bachelor, but Cheer was legit, because it wasn't contrived for TV.

I wish there was more of this on TV/streaming services. Unless you're a robot, documentaries are an instrument of empathy creation for your fellow humans. I think that's valuable. It's also the kind of thing that I aspire to. In high school I followed around a cheerleader during tryouts to show what that process was like, and that aired on cable access. In my first real job, I followed around high school freshman to show what it was like to start high school in an over-crowded school. I did a mini-doc after the fact to talk about the creation of a roller coaster. It's my thing even if I haven't pursued it on purpose.

The upside of the streaming-ization of TV is that there's more of this available than there used to be. That's exciting. I love documentaries.

If you've been to Epcot at Walt Disney World, you've probably seen The American Adventure, the audio-animatronic show that chronicles the history of the United States. The thing that I love about that attraction is that it's a surprisingly brutally honest review of our history. It doesn't gloss over slavery, the Civil War, the destruction of Native Americans or the wanton destruction of the environment. Despite all of the negatives, somehow the great experiment persists.

I always leave that attraction simultaneously proud, sad and hopeful. There's a montage at the end, to the original song "Golden Dream," where they show so much of our achievement, including shots of Space Shuttle astronauts, Dr. King, Kennedy's "Ask not" speech, famous Olympic athletes, Steve Jobs, Peter Jennings, etc. Our history is legitimately dark, but we've slowly been able to overcome some of it. It makes me want to be a part of the generation that helps us move above our worst.

I was a product of the tail end of the civil rights movement, specifically the part where desegregation was a forced issue in the schools. If environment is a deciding factor in how our views are shaped, I'm thankful for that experience. It's not that I don't see color because of it, it's that I see that color is not the determining factor in how people are valued. To not see color is to ignore the inequality that persists.

We were watching The Help today, and so at dinner, I had to explain to Simon what racism is since he wandered into the room while it was on. "Those people were stupid," he said, even though we're trying to discourage the use of that word when describing people. If it's really going to take another generation to shake racism out of our culture, hopefully I'm doing my part.

I desperately want that optimistic version of America in that Epcot attraction.

I had a great conversation today about content creation, and all of the various forms that takes on. Sometimes I forget that I had a previous career where writing, audio and video creation were my primary function. If I were to put a date on my flip to software developer, I'd probably put it in 1999, so I've been at that for 20 years. Since the flip, I have to remind myself that I wrote a book, a technical book, which is really hard and few people do that. Writing a book is like getting a tattoo. It'll always be there, and you'll always have a tattoo, er, be an author. I did over 200 podcast shows before podcasting was cool. I've squeezed out short documentary video pieces. I think my total spend on gear since 2006 has been nearly $15k (yikes, expensive hobby). I've written thousands of blog posts. I've also published a ton of stuff on that Internet, amateur design and what not.

All this to say that most of this content creation has come naturally, with joy and without any real planning, outside of the book. Recently, I've really enjoyed researching and writing the CoasterBuzz 20 pieces, seeing how much things have changed in that time, and how silly a kid I was. (Now I'm a silly middle-aged guy!) I've been watching documentaries and various Masterclasses from movie and writing people, and that has me energized. I actually opened Photoshop in the last two weeks, though I can't remember how to use it as effectively as I used to.

The act of creation has always been at the core of what I do, and I've come to realize that it's a core attribute of who I am. It's an intrinsic motivator. It's on a short list of things where I have to ask myself, "Is that I'm doing feeding this thing?" It applies to so many aspects of life, including parenting and professional endeavors like writing code and helping others develop their careers. Creation leads to outcomes.

There's something satisfying about realizing that I never truly left that previous career. I mean, I have a Discovery Channel credit for video!

I was watching something on TV talking about how social media is such a one-sided view of a person's life, made worse by the fact that some people use it as a means to build a persona that is impossibly unrealistic. That kind of bothers me. It's not really that different from real life. You rarely encountered water cooler chat before the Internet where people would describe their weekend as "terrible and here's why."

Keep in mind, I'm not talking about the Negative Nancy personality, which is intolerable in real life and on social media. That person who is always complaining about innocuous things that suck (coffee, lines for getting coffee, traffic, rain, etc.) is not fun to be around in any circumstances. But the person who is enduring hardship, or struggles with depression, we tend to have a cultural contract that expects us to repress these things. We need to figure out how to change that. Any time I've written about such things here, and it's rare despite the frequency, it has been met with empathy and "me too" reactions. Clearly the need to share is real.

Team Puzzoni has collectively been taking a beating since the year started. All three of us have had more than our share of difficulty, bad news, conflict and just general bullshittery. Some of the challenges are transient, some of them have long-term impacts, and most frustratingly, few have really been in our sphere of control. The last month has been exhausting.

Fortunately, we've got each other, and that's a big deal. It's still frustrating that it isn't cool to allow more bean spillage with the prospect of getting a random "you've got this" from a stranger. It feels like we live in a world that generally lacks empathy and people are all suspect of each other for a hundred stupid reasons.

So that's my release. We need a break, and I've said it out loud. Here's to better days going forward.

I am a huge, huge fan of Masterclass. I know it isn't cheap, but it's worth every dime. The instructors are experts in a great many fields, not just creative, but also in business and science. The intent is not to become experts or achieve the level of the instructors, it's to have access to the wisdom and knowledge of people you wouldn't ordinarily get to meet. What you do with the information is entirely up to you.

This is the reason that we go to conferences in the software world. Being around the same people all of the time can make you dull. That's not a slight against them (except when it is), it's just what happens. Knowledge and wisdom comes from experience, and it's the one thing in life that you can't just naturally obtain independent of your environment. You have to "see the world."

We naturally gravitate to these learning experiences in our spare time, which if you think about it, reinforces that one of our greatest intrinsic motivators is in fact learning. Diana didn't know anything about quilting ten years ago, and today she wields a $10k long-arm machine like a boss. I'm that way with photography, and it could not have been a thing without learning from a great many resources.

It's my belief that as business leaders, we should take ongoing education more seriously. Admittedly, my line of work suffers from a serious gap between experience and need, so I might care about this more than practitioners of other fields. But in any business, it stands to reason that investing in your people in a non-trivial way is a thing where everyone wins. You demonstrate commitment to your people and they get better at their jobs. There's that old legend where a CEO asks the COO, "What if we teach them and they leave?" and the COO says, "What if we don't and they stay?"

Education, experience, learning... these are all undervalued in our culture even though they solve so many problems. We need to stop missing these opportunities. Critical thinking and observable evidence combined with an understanding of history can go a long way toward making us better humans.

Building multi-tenancy into your app is an interesting (and dare I say fun) problem to solve, because there are a number of ways to approach it. And now that we don't have to spin up closets full of hardware in some basement, there are better options that we didn't have in the dark ages. I'll talk a little bit about the options that I've used, and how I solved the problem this time around with hosted POP Forums.

Establishing tenant context

To start, let's define it: Multi-tenancy is where an app has to accommodate a number of segregated customers, where the data doesn't overlap. This is different from just having multiple users, because while there are overlapping concerns between your users, they're all associated with a root customer. Think about the granddaddy of SaaS apps, Salesforce. When you're using it, you don't encounter the data of Company B down the street, only Company A, where you work. You're potentially using the same hardware instance, and definitely the same software, but you see different data. That's multi-tenancy.

Fundamentally, I believe your app needs what I call tenant context. Regardless of what part of the app is acting on something, whether it's a web-based action, API, some serverless thing (Azure Function or AWS Lambda), etc., it should know which tenant data it's working with. Setting the context is the first step of any action. At the web end of things, this is straight forward enough, because you could operate based on the domain name (customer1.contoso.com), the path (contoso.com/companyA/home), the query string (that's so 1999) or whatever your authentication and authorization mechanism is (auth tokens, for example). Some asynchronous thing, like a serverless component running in response to a queued message, may have tenant context baked into the message.

Partitioning the data

Once you have tenant context established, you have to decide how your data will be divided up. The strategies here are about as diverse as the persistence mechanisms. Many no-SQL databases already partition data by some kind of sub-key or identifier. The easiest way to isolate tenant data in a SQL database is to use a compound or composite key on every record, where you combine tenant ID and a typical identity field as your key. If you use Entity Framework, there are even a number of ways and libraries to add plumbing to all of your queries and commands to specify the tenant context, often by having your entities derive from a common base class that has a TenantID property. You could also divide up tenants by schema in SQL, or just provision a totally separate database entirely for each tenant. That's less scary than it sounds when you automate everything.

I've had to personally solve this problem several times over the years, and I never did it the same way twice. However, this allowed me to make some educated decisions when it came time to do it for POP Forums. Architecturally, the app consists of two web apps, one that serves the forums, and one that acts as the management interface for customers (billing, usage reporting and such), while a bunch of Azure Functions do a great many things in the background, triggered by timers and queue messages. The databases are SQL, using Azure SQL Elastic Pools.

Running code, in context

Let's walk through these parts. If we go to the source code, you'll find that the application is technically a monolith. The web app and Azure functions share code and they share the same database. (Moving compute around doesn't really break down your app into "microservices," because the parts still need each other and share state via a common database.) There is an interface called ITenantService, and the default implementation wired up in the dependency injection has no implemented SetTenant(tenantID) method. That's because if you're cloning this app and running it as is, there are no tenants.

Hosted POP Forums has two replacement implementations. For the web app, the SetTenant(tenantID) method puts the tenant ID into the HttpContext.Items collection. The GetTenant() method fetches it from the same place. (If you're wondering, there's a great new interface you can add to your container called IHttpContextAccessor which will get you to HttpContext, making all of this stuff easy to test.) Middleware placed early in the Startup sequence uses this implementation to set the context based on domain name. The web app also has a replacement for POP Forums' IConfig interface, which normally just reads values from a JSON config file. The replacement looks at the tenant context and builds up the right connection string, which is in turn used by all of the default SQL code. Obviously there is some error checking and redirecting for bad domain names and such, but that's (mostly) it for the web app.

The Azure Functions have an even simpler version of ITenantService. Since the functions are one-time use and thrown away, it gets the tenant ID from the queue message, stores it in the instance and reads it back as necessary. Relative to the open source code, there are not a ton of replacement implementations in the hosted version of the app. The biggest one writes error logs to a common instance of table storage in Azure instead of writing to the database.

So what about the timer based functions? The timers each queue messages in a fan-out pattern to run each task in tenant context. For example, one of the tasks is to clean up the session table, which feeds the "users online" list on the forum home page. It just nukes rows where the last updated time is too old. The timed function queues messages composed only of a tenant ID, one for each tenant. The listening function runs against that tenant.

Design rationale

One of the biggest requirements when I started the project was to modify as little as humanly possible from the open source project. I'm reasonably consistent about doing something with it on a weekly basis, and I don't want to change a consuming project beyond updated project references. So with that in mind, that's why I went with separate databases. Elastic pools make this economical and super easy, and when (if) I hit the ceiling for number of databases in a pool, it isn't hard to add some little bit of context about which "server" to look at for databases in the new pool. The downside, yes, is that any schema change has to be made in every database. Still, that's historically a small blast radius, and using something like DbUp, I could likely update hundreds of databases in a few minutes.

The domain name as key to defining the tenant is just obvious. People like to name things, and with additional automation, I can even provision custom domain names and certificates.

I've done some initial load testing in an unsophisticated way, and with two S1 app service nodes running on Linux and the old-school 100 DTU database pricing model, I can hit about 2,500 requests per second with sub-second response times. The unknown here is that my tests aren't hitting huge datasets, so the cache load (there is two-level caching going on with Redis and local memory) is not large. I'd like to design a better test and try P2v2 app services with three nodes, and flip the database to the vCore model. I think I'd need some impressive customers to really drag it down.

Overall, multi-tenancy was the easy part of the project. The more interesting parts were the recurring billing and provisioning. This week I'll be migrating one of our big sites on to the platform.

One of the things that I really liked about college was that, for a school in the middle of Nowhere, Ohio, there was a reasonably strong international student program. The residence life program, which I was a part of as a resident assistant for two years, also recruited heavily from that pool. After going to a high school that was essentially all white and suburban, it was my first exposure to a bigger world. It's also one of the reasons that I grew particularly interested in the people of India and their many cultures, because one of the other RA's in my building was Indian.

When I rolled into a software career a few years later, being around people not born here, either by way of immigration or work visa, became my default. Having not really sowed the seeds of travel desires until after I became a parent, I haven't had the opportunity to travel abroad as I'd like, so these folks have become my next best thing for now. I am fascinated by their stories, about what is different, and what is the same. I've made great professional friendships with people from India, Pakistan, Japan, Korea, China, Macedonia, Serbia, Croatia, Russia, Uzbekistan, Ukraine, Syria, Turkey, Germany, UK, Ireland, Australia, Mexico, Dominican Republic, Columbia, Venezuela, Brazil, Chile, Costa Rica, Argentina... and certainly others that I've forgotten. Oh, and Canada, eh?

As you might expect, this is one of the reasons that I love cruising as well. The cruise lines bring together people from all over the world who work their asses off for you. I enjoy the opportunity to talk to them about their families and the homes they go for months without seeing. My only wish is that they could spend more time being social and less working!

About three cruises ago, we had a server in Palo, one of the nicer upcharge restaurants on the Disney ships, who was unusually young, presumably 22 at the time. I say unusual, because it's typically the best servers who work up there, and excellence does come with experience, and therefore age. She was from Estonia, but most recently living outside of London. She was charming, smart, interesting, with the sort of kindness that you can't really fake. To that end, we requested to get her serving us the next two cruises, and were successful the last time. From there we followed her on social media. She mentioned that she would be passing through Orlando after her contract on the way to Columbia and needed a place to crash for a little less than 48 hours, so we offered. It seemed unlikely that she was an ax murderer.

We were pretty excited because it was a little like having an exchange student, only post-university age and free enough to travel the world. Her perspective on the world, and really life in general, seemed 10x more informed than my own when I was 23, and in general she was a lovely human being. Simon had a big sister for a day or so. It was actually the first time she ever spent time in the United States, in an American city, since to and from the cruise ship she never had the opportunity to move around. Hopefully we represented well, taking her out to the legendary Hamburger Mary's (between drag shows, unfortunately), and the timing was right to get cheap tickets for Aladdin at our beloved Dr. Phillips Center. Also, she was definitely not an ax murderer.

Our next international encounter will be with family friends from Norway this summer. That's very exciting. I might not be able to travel as much as I'd like to, but it's great when a few people can come to you. The world is a beautiful and diverse place, and I value the opportunity to meet the people in it.