The explosion of creativity and art online

posted by Jeff | Wednesday, April 8, 2020, 10:30 PM | comments: 0

The isolation of the pandemic has been rough probably for everyone, even if they've completely avoided sickness. But something extraordinary has happened because of it. People are going to great lengths to be creative, and make art. Certainly famous people are doing it, but so is seemingly everyone else. It's the phenomenon that I often quote Jack White with illustrating (in the documentary It Might Get Loud), that constraints can in fact force a great deal of creativity.

I'm not shy about saying that so much of what's on the Internet, produced by amateurs, is terrible. But people are trying harder, and the good stuff is rising to the top. Best of all, there are more cats, dogs and toddlers. Heck, even the pros, especially the late night hosts, are figuring out how to wing it from home, which is pretty cool.

The Broadway community has been doing some really great stuff. Caissie Levy did this great acoustic version of "Let It Go" (she's the definitive Elsa, by the way, and I'll stand by that having seen her do it). The original cast of Hamilton got together to perform for a little girl who was supposed to see the touring show, in the weirdest way on a "show" that Jack Ryan, John Krasinski is doing. (Also, he's married to Emily Blunt? I didn't know that.) Ben Platt and Dear Evan Hansen-folk did something similar for "You Will Be Found." And that all was just in the last week. Broadway performers and their crews are not rich (not most of them, anyway), and much like our theme park workers here, they're hurting with all of the theaters shut down. I give them a lot of credit for finding ways to do a show anyway.

But even here in the suburbs, sidewalk chalk drawing has become a prominent way to communicate with your neighbors. One of our neighbors made wooden "eggs" for kids to decorate. And for the kids not going to school, they're doing stuff from home as part of their classwork.

Art, and especially music, is something I find myself going deep into lately, in ways I haven't in years. It's what I need to mediate and relax. Seeing live music, and musical theater, are things that bring me great peace, and not having that access is hard. But I'm hitting the archives in ways that I haven't in a long time. I can't play anything or sing, and by extension can't write music, and maybe that's why I started a radio show (it's huge in Alaska!).

It's hard knowing all of the people we planned to see this summer, like friends and family in Seattle, or hosting friends from Norway, won't happen. My optimist side, which has taken a beating, is trying to view this all as an opportunity to make things. It seems like I'm not alone.


Fake tattoos

posted by Jeff | Saturday, April 4, 2020, 3:40 PM | comments: 0

We've been having fun with fake tattoos from Inkbox. They have a ton of fun designs, and they last about a week and a half, depending on the "traffic" to your skin in that spot and whether or not it's generally moisturized. You can also upload custom designs. The newer ones have some pretty good shading, too. I totally loved the spaceman. I did a custom one with a lyric from the song "Sound" by the band James, one of my favorite songs of all time. We also created some anchors with Mickey ears that would be fun for our next cruise, but that's obviously on hold.

Yeah, it's probably a little midlife crisis-ish, but I'd like to get a real tattoo. When I was younger I was too non-committal about such things, but meh, at this point in life it doesn't really feel like there's anything at stake. Diana got her first tattoo at the age of 30 (of a cat, natch), and she'd like to get another one too. I can't really describe the motivation or intent. Even after seeing the Jeff Goldblum episode about it, I'm not sure what the intent is. I've been around tattoos for my entire adult life, as my first wife Stephanie had her first before we met.

For a long time I've been fascinated with the endless symbolism associated with a navigation compass rose, and figured some artistic take on that would be a starting point for me. In recent years I've also loved the mandala style of design, and I'd love to see some combination of the two, the stark angles against the curves. Diana has some ideas to combine the quilting "feathers" pattern with other shapes, so that's where her interest is. We found a guy here in the area that does some pretty amazing line work, which is what we're looking for. He's generally booked months out, but as you would expect, had to stop. He does text really well, too, and I wouldn't rule that out. Some day, we'll be able to get real.


Random pandemic thoughts, debut edition

posted by Jeff | Wednesday, April 1, 2020, 11:48 PM | comments: 0

I've decided that I need to brain dump now and then, as the pandemic works itself out. I'm experiencing epic new levels of anxiety, as I'm sure most people are, but not everything I think about can be distilled into a witty blog post. And what I write, I can at least temporarily retire, so that should help with sleep. Also, I want to have a record of this, so I can have a laugh or cry about it at some future date. So here's the first of such brain dumps.

  • Simon and I are at 14 days since our last world outing, which is to say having visited a place where we could have potentially been exposed. That trip was to Winter Garden's Plant Street, where everything was open as if nothing was going on. We bought Diana a bike. That area has since been identified as a hot spot by the county mayor. Diana is 10 days since last potential exposure, when she had to go in to Publix because you can't guy Sudafed through the pharmacy drive-through.
  • Much of my anxiety has to do with the fact that COVID-19 is unkind to people with a history of respiratory issues. Diana has had many throughout her life, including bronchitis last summer. Simon has had pneumonia and had to use a nebulizer for meds twice in his young life. The idea of them getting the disease scares the shit out of me, especially as we head into the inevitable ramp of cases in our area. Our most recent doubling, to 450 cases, happened in about three days.
  • I've stopped watching TV news. I can't. I'm keeping up via the New York Times, which has been brilliant, but even there I skip a lot of headlines about the volume of death.
  • It's amazing how our neighborhood of 300-ish units (townhomes, bungalows, McMansions, lakefront homes) is filled with people being outside, every single night. They weren't properly social distancing, but it seems like they're all doing it now, including the kids, fortunately. We had a few days of record high temperatures, where 95 is still 95 even with low humidity, but today was an amazing 78 degrees and sunny, all day. I never thought living in Florida would be advantageous because you need to get outside when you have to isolate in a global health crisis.
  • I said this before, but grocery delivery has to transition from a luxury to a normal thing. The efficiency is staggering from a transportation and carbon footprint standpoint. One car covers a dozen places instead of a dozen cars going to one place. It's also essentially a zero-contact operation for the drivers, if they're doing it right at the dispatching warehouse. My beef with the way that Amazon does it is that they're all gig workers (or they were), which means they don't have health insurance in a time when you damn well need it.
  • Delivery is more expensive, but it also lacks certain things that we'd like to have or need. The weird things are that we can't get Egg Beaters (or generic brand), basic Ritz crackers, V8 Fusion juice and some other minor things. I'm the only one who eats the 'Beaters, because it's a low-cal way to make a breakfast that I otherwise load up with hashbrowns. Still, I really want to avoid grocery stores for the next four weeks at least. Our case count doubling in Orange County is now down to three or four days. The storm is coming.
  • I feel for the teachers trying to remote teach, and clearly it's hard when not all kids even have computers. Simon's two teachers are doing things a little differently, and it's harder to prioritize what one is asking for because of the way she's using the online tooling. And how do you even approach satisfying an IEP when the playbook as it was known two weeks ago is thrown out the window? Diana is feeling enormous pressure here, when normally he has two teachers, an ESE teacher, a counselor and a staffing specialist all looking out for him and his education in person. They're doing their best, as is Diana, but it's a lot of chaos.
  • Today we had a meltdown because we decided to allow some suffering instead of rescuing. It was a simple enough circumstance: We were playing frisbee and Simon wildly through it and it rolled into the neighbor's bushes. To get it, he had to duck under a small palm tree, but he wouldn't do it. After 15 minutes of asking us to get it for him, Diana did and kept it, and we were the worst most hated parents in the world for the moment. If there was a good thing, he did a great job of describing his feelings about it all whilst expressing his distaste for us as parents.
  • I started a new job today. It might be the first time ever that I've considered work a useful distraction from life instead of the other way around. First days are usually not super productive, but I'm optimistic as usual about the gig. Lots of smart and experienced people, and the culture seems to largely have survived the rapid growth.
  • It's already Wednesday and I haven't contributed anything to GitHub yet this week. I have a streak going of nearly 16 months. Need to work that in.
  • I want to get takeout food from a local restaurant, but it's unclear which ones are really doing proactive no-contact hand-offs. I worry about those businesses.
  • It's very telling that Disney hasn't even bothered to set an intended opening date. I don't know how you open theme parks in this situation. Disney will be fine in the long run, but I'm pretty worried about the people that work there and associated businesses. That's a huge chunk of the population in the Orlando metro not working.
  • With the concentration on the pandemic, I'm worried about what's going on while we're not looking. Idaho passed a bill that discriminates against transgender people (the courts have already ruled against similar measures). The White House is completely rolling back fuel efficiency standards. It seems like there is enough to be vigilant about to have to add that stuff too.
  • Enjoy this distributed performance from the cast of Dear Evan Hansen. You know the song. You need this right now.

It's OK to be a mess, as you see fit

posted by Jeff | Wednesday, April 1, 2020, 9:25 PM | comments: 0

One of the hardest things for me as a parent is to remember that the things that cause your child tears really are the worst things ever for them. That's why the old cliche thing we used to hear about eating your dinner because kids are starving in Africa was a pretty pointless thing to say. We had no context at that age. And boy, layer on autism in that equation, that introduces all kinds of challenges. The point is, your kid's reality is what's real for them.

I'm often astounded at the way that adults treat each other, in the context of our individual realities. I'm not debating whether or not something is objective fact or not (don't even get me started), but rather that our lives can be difficult, and everyone has something. I'm not sure why people are so certain that anyone does not have something. Sure, it's human nature to measure people up and draw some judgmental conclusions, but it doesn't make it OK. I feel like everyone is broken in some way, and to various degrees. Even if someone is by every socioeconomic measurement "privileged," it hardly means that their life is free of pain.

What's worse is this horrible and toxic call-out culture in social media. People really love those virtual high-fives and likes when they tear someone down for not being empathetic or woke enough. I'm not sure why anyone thinks they're changing minds, but they seem to feed off of their righteous tear-downs. As it is, social media representations of people are at best heavily filtered, and at worse completely fictional. You don't know the person on the other side of the screen. Just back off when someone decides to share a little pain or anxiety. You're not the score keeper or inventory master for all of the karma in the universe.

You're going to be feeling a lot of anxiety, anger, sadness and God knows what else the next few months. You may even feel bad that you feel bad, but you need to stop that nonsense right now. Every therapist will tell you that it's inappropriate to measure your pain against someone else's. Therapists will also tell you to avoid relationships with people who attempt to invalidate your feelings, so don't spoil that relationship with yourself. What you feel is as real as it can be, and measuring it against someone else's feelings will not enable you to process them.

It's OK to be a mess. Express it as you will. You don't have to feel bad about it.


I'm sort of doing a radio show... in Alaska

posted by Jeff | Monday, March 30, 2020, 11:45 AM | comments: 2

My last time doing an on-air shift at a commercial radio station was at the start of 1996 (Cleveland's Shit Music, Jammin' 92.3!). As the story goes, and I've described it many times, I was doing overnights with no contract, and the terrible midday guy with a contract got booted out of his spot and they had to stick him somewhere or pay him out. I was handily moved to part-time. With the realization that I was "losing" a $15k salary and still living with my parents anyway, I decided that was going to be the end of my radio career. It was a terrible business.

I really liked playing tunes for people though, "Macarena" being a notable exception. It wasn't like real work, and that's probably why it didn't pay like real work. I got to that job so quickly after college that I let my ego get the best of me, thinking it was a legit way forward. But it was a lot of fun.

In 2009, right before we moved to Seattle, I did a couple of radio shifts on my college station. Gone were the days of it being on the air 18 hours per day, and they invited me to drop in, fire up the transmitter, and do whatever I wanted. I did what any wash-up from 13 years prior would do: Play all of the music I never got to play. I did a mix of alt rock spanning almost 20 years, and it was super fun. Ten times as fun because I didn't have to make a living doing it.

It seemed absurd to me that there wasn't a technological way to achieve this legally today, and as it turns out, there is a way. A friend of mine that I've known for a long time via the roller coaster community has a show called Spare The Rock, Spoil The Child, and it airs on public radio stations all over the place. His home base is currently KUTX in Austin, but he can just as easily do it from home. He uploads the show to PRX, which acts as a distribution mechanism to radio stations all over, and you can stream shows there. PRX and the radio stations handle all of the licensing, so you don't have to.

I decided to make a show called Modern Gen-X. The idea is simple enough... play alternative rock from the last three decades and mix it in with current stuff. No one is really doing this in commercial radio, and I haven't been able to coax a streaming service into it either (playing Billie Eilish and Screaming Trees is apparently incompatible). Maybe no one actually wants this, which is also possible.

I don't think that very much radio is really live anymore, but for me, much of the joy in radio was playing the songs. I've endeavored to do that, instead of just editing a show together. I play it "live" and mix as I go, and if I choke on words, and I do because I haven't done this professionally in 24 years, so be it. I want it to be fun and in the moment, and for an hour recapture those feelings of college radio.

There are two shows in the can so far. I don't have any production value yet, which is to say there are no stabs or anything from a voice-over guy. No theme song. I suppose I may get there eventually, but for now, it's fun to just play the music. I bought a cheap DJ controller so I at least have physical buttons to push, and despite having done a podcast back in the day, I never did buy a good microphone until just now.

PRX allows you to charge for your program, but I have mine set up to be free. So far, a station in Alaska has "bought" it, which is to say they may air it. That's kind of neat. I'll keep at it, and see if others follow. It's a fun distraction, and if I commit to it, it forces me to take a break and listen to music, which really helps me chill out. We can all use a little more chill right now.


Transformational outcomes

posted by Jeff | Thursday, March 26, 2020, 12:00 AM | comments: 0

In the midst of a global pandemic that is killing thousands of people and wrecking significant parts of the economy, it's pretty easy to devote a lot of anxiety to all of it. With my growing interest in anthropology and history though, my mind often wanders to what the world looks like on the other side. This is, to an extent, a self-defense mechanism, because I'd much rather think about how things improve than think about the odds that one of us don't make it.

Healthcare, naturally, will have to fundamentally change. Outside of the US, there is already a lot of talk about "surge capacity," which can be a thing in systems that are not designed for profit. Here in the US, we're seeing the worst combination of our problems: healthcare is linked to insurance which is linked to employment. That's a scary thing when unemployment is hurtling toward 1 in 5 people. There is also emerging evidence that survival rates, and early on even the ability to be tested, were largely tied to income levels. There are a whole lot of moral implications to consider, as well as practical. As disease is unconcerned with socioeconomic status, containing transmission becomes harder when a segment of the population doesn't have healthcare access. At the end of the day, my hope is that we'll finally have the conversation about why we spend more per capita than most countries but have less positive outcomes. It's an objective truth we've ignored and done nothing about for a long time.

With all of these people working from home, I imagine that businesses who operate largely with desk jobs will ask some hard questions about whether or not they need real estate on the books. My perspective is admittedly skewed about this, because I've worked mostly remote for years, and even now, my org is 90% remote most of the time. Smaller companies in particular take on some volume of risk when they sign a lease. And then you get annoying things like last year, when our building air conditioning died. Still, I see the appeal of co-locating, and understand why people enjoy it. If it weren't for the commute, I might even prefer it some of the time.

School was already starting to transform slowly, as more and more districts send devices home with their kids. You can do so much now with a cheap Chromebook. And it's important that the school provide the hardware so as not to discriminate against the poor kids who have no say in their socioeconomic status. The underlying software is a huge mess at the moment, which I can see as frustrating, for teacher, parents and kids. There was already an industry around distributed learning, but I can see districts taking a hard look at it now in a broad sense.

To minimize exposure to other people, it has made sense to order stuff online and have it delivered, including groceries. This is unfortunately still priced as a luxury, but think about the extraordinary efficiency it involves. If 10 people on the same street all go to the grocery store, they're all (unless they drive EV's) burning fuel and emitting CO2. But if one person delivers those goods to all 10 of those people, only one car had carbon impact. And someone now has a job, whereas the 10 individuals all have their own job already. That sounds like an opportunity to me, so how do we make it scale?

And of course, we'll have to look at how we handle this kind of global health emergency. In fiction, the zombie apocalypse always comes about because the disease spreads too fast to be contained, and there is no coordinated response. We're seeing that first hand, as leaders at every level in the US are in a state of decision paralysis, incapable of critical thinking or understanding science, more worried about the political implications than the outcomes. Everything that has happened has happened according to predictions from scientists and experts. The US even had a head start, seeing it happen in other nations weeks ahead. There's a whole lot of failure to go around, and we can't pretend it didn't happen.

If there's anything though that I hope we'll look harder at, it's objective, observable truth. Even in the context of the pandemic, I'm horrified by people who refuse to engage in any critical thinking, and reject what they can plainly see in front of them. Sports rivalry politics and tribalism has caused people to lose their minds. To draw a comparison, I'm originally from Cleveland, and I'm used to people sticking by the Browns year after year, even though for my entire life, they've never won a Super Bowl. But the thing is, nobody dies when the Browns suck, which is every year. Applying that commitment to the political party of your choice is insane. The older I get, the more socially liberal I get, and fiscally conservative-ish I get, and neither party really checks both boxes. (The GOP doesn't check any boxes at all... they're creating the worst deficits and national debt of any party in history.)

A segment of the population has to stop granting politicians, especially the president, the flexibility to mix fact and fiction. It does not serve us. "Alternative fact" is not a thing... it's either a fact or it's not. Right now, the difference between fact and fiction literally costs lives. We can't operate like that.


Team Puzzoni in quarantine

posted by Jeff | Saturday, March 21, 2020, 12:05 AM | comments: 0

You've probably heard that there's a global pandemic, and if you're responsible and engage in even the least bit of critical thinking, you're at home right now. Saying that these are strange times doesn't quite capture it.

This last week was going to be Simon's spring break anyway, but it was originally to start with a 3-night cruise, followed by a possible audible to Washington DC or New York. Isn't it funny how absurd that sounds now when it didn't two weeks ago. We made a pretty late call on cancelling the cruise, because there was no useful guidance from the federal government, and even the theme park closures and general shut down didn't come to be until after the boat left. We started to look at alternatives, but even a quiet night or two in a Florida hotel quickly became an obviously bad idea. What's remarkable about that shift in thinking is that it happened in four days, not weeks. I suspect our story is not unique.

Our mental health

We were struggling a bit even before this COVID-19 stuff got real. Last week I described our challenges with Simon and his meds, plus the reduction of video game time. We started the break Friday with an bedtime meltdown, and went 2 for 3 on the weekend. Since then, there's been some anger issues, but not nightly meltdowns. We even had two relatively good nights. As soon as it was clear that school would be delayed, Diana immediately came up with a schedule, which hasn't been strictly enforced because there's formal learning coming from the district, and honestly he hasn't been receptive to it. She found some great alternatives to the officially prescribed online stuff, so she's working that angle. Overall, we're still seeing the anger, but family bike rides and frisbee and no games in two days might be having a positive effect. We just don't know how to find his baseline. I think he's "better," but it's hard to quantify, especially when he still insists he "can't" do the simplest of tasks and gets angry.

Diana is naturally out of work because the theaters are out of commission indefinitely. The tours are all grounded. She's assumed the role of teacher and while I admire her patience, I know she has limits. I haven't helped as much as I should for a number of reasons (I'll get to that). As is her personality, she's taken to cleaning, deeply, all over the house. We bought her a bike, her first new bike in her entire life, so we can ride around a bit once it cools off. It's therapeutic for all of us, and it's striking how many of the people in our neighborhood are out (social distancing, of course).

As for me, I've been struggling on and off. I mostly completed the things I hoped to do as "sabbatical projects" by the end of February, and so I've been getting stir crazy. Pile on the Simon challenges. I was waiting for a very long time for a lucrative contract gig to start, but then had a permanent thing come up and another more casual career conversation in play. That all landed and became clear and I'll go back to work on April 1. In the mean time, with the pandemic stress factored in, my IBS has been like torture, and allergies woke up this week too. It's hard to evaluate your physical health in those circumstances.

But I'm learning to be present. I did that today. I spent a few hours with Simon building Lego, I caught up on some technology news, looked at my operational stats for the forums and sites, and gave Diana a few extra squeezes throughout the day. And yeah, that bike ride.

Then there's the fear

I've spent probably too much time reading the news, but I've been sticking mostly to the New York Times and Ars Technica. I'm frustrated that I had to really come to the conclusion about the severity of the outbreak, and the correct actions, on my own. Those four days over last weekend involved as much reading and learning (and bullshit avoidance) as I could muster. I'll never be an epidemiologist, but I know more than I did a week ago. And what I know scares the shit out of me, in part because so many people haven't got the message I have.

Financially, we're not in a scary place, but the reality is that short of being independently wealthy and loaded with liquid assets, no one can really be sure that they're "stable." I made that mistake after 9/11. I suspect the long-term result of this for me will be a desire to hoard money, which isn't good for the economy.

If one of us does get sick, then what? Are we able to get tested? Is there any point in trying to isolate from each other within the house (if it's Simon, that's obviously not possible).

The disease itself doesn't scare me too much, though the odds of death are way too high to be comfortable. Diana's allergies and last year's bronchitis, Simon's pneumonia some years back, I don't know how vulnerable they are. And even then, it's the overwhelming of the healthcare system, and its dependencies on insurance and employment, that I worry will wreck our society. We live in a place where a huge percentage of the population relies on a thriving service and hospitality industry, and we're going to lose a significant portion of the 75 million people who visit annually. Our tax base is what it is because of the tourists. I realize it's too early to see how that shakes out, but it's not fun to think about.

My most immediate concern is the continuation of stocking the pantry. There are no indicators that the overall supply chain is at risk, but grocery shopping will not be the single riskiest thing that we do. There's no protocol for making that work in a safe manner. I don't even think anyone is thinking about it. Every cashier is now a likely infection vector, putting them at the level of healthcare workers, in terms of importance to our survival. Forget delivery services, because many of them are already overwhelmed.

Our contagious clock

We went out Wednesday morning, primarily to buy Diana that bike. It was wholly necessary, but that resets are contagious clocks. People in downtown Winter Garden, where the bike shop is, seemed to not be taking it all very seriously. We also picked up some Thai takeout, because I want to support the local restaurants.

So we're potentially "hot" until April 1, and that's only assuming we don't go out for groceries, which one of us will likely will have to do. I don't know what the protocol for that is either, but I assume if one of us goes out, we've essentially all gone out. I worry even about our neighborhood. One neighbor is well over 70, the other had an immunocompromised child, there are pregnant women in the neighborhood and so many parents are letting their kids mingle with others. I don't want them getting sick because of me, even if we're doing our best to stay away from each other.

What we can do

With all of the focus on the chaos and lack of control, I suppose there are some things we can do. Cleaning around the house certainly isn't the worst idea. We're breaking out Lego we haven't seen in awhile. We're strangely more active. We can celebrate the death of standardized testing in Florida, for this year at least.

It's hard to volunteer for stuff when you can't go anywhere, but we're looking at things to do online that benefit others. Heck, I've got people making pull requests to my open source projects! We're also putting money in places we might not have thought about in "normal" times, ranging from organizations to individuals we know and can see how we help.

The optimist in me believes that there's an unprecedented opportunity for us take a hard look at our society and culture, with humility and honesty. With that look, perhaps we can improve ourselves.


So you're working from home for the first time

posted by Jeff | Monday, March 16, 2020, 1:00 PM | comments: 0

If you're fortunate enough to have a desk job, there's a good chance that you're working from home today. If you've never done it before, it may seem weird, uncomfortable and strange. I've worked remotely for roughly 6 of the last 8 years, and I find remote work to be far better for mental health, productivity, work-life balance, and I definitely prefer it. There are negatives, certainly, but it can be a net positive experience. Let me share some of mine.

Now, keep in mind, literally no one has ever done this during a global pandemic, so there are aspects of this that are not in anyone's experience locker. Everyone is also enduring a sense of helplessness, anxiety and worries about things no one was thinking about two weeks ago, and you're dealing with that in an isolated way. I have no idea how to roll with all of that. All I can do is account for the physical and cultural aspects of remote work.

Let's start with the physical environment. You can't work from your bed or on the couch. Seriously. One place you sleep in, the other you melt comfortably into a blob to passively do leisure things. You have to sit at a desk or a table or something and be upright. You need a good chair. If you can do it, plug in a big monitor and keyboard to your computer. Really, just think about where you work in the office, and try to have that thing. I'm not saying you never go out to the patio for a spell to break up the day or change perspective, because I've certainly done that in offices with break rooms, beanbag chairs or whatever, but mostly work from your desk.

If you have a door, and you can close it, you should. This is the part that's going to be extra hard if you have kids because they're going to be home. You might just need to be flexible, and ask for a little flexibility from your coworkers. In my case, my wife works for a theater, which means that she's not working, so we're "lucky" I guess? Even still, we have a child with ASD and ADHD, so we have a schedule for him because he can't roll with a totally unstructured day. There's intermittent free time, "phys ed" time, computer-based learning from school, reading, etc.

Speaking of schedules, have one. There was a point where I fell into a trap of over-working, which is to say that I spent more time at my desk but not necessarily being more productive. Countless studies conclude that there is a limitation to productivity for most people, and that more time does not result in more output. I used to roll out of bed at 7, be at my desk by 7:45, and not leave it until 6. I didn't really stop at lunch, I just ate as I worked. I was going at it for 10 hours a day and burned out. My boy would knock on the door at 6 and I'd feel bad that I hadn't stopped yet. Have a stopping point and walk away. Break for lunch and do something else.

Almost every organization has the ability to do video calls now, and you should use video whenever possible. Hangouts, Teams, Slack, Zoom, Skype, etc., everyone has this stuff. If you have a noisy environment (city dwellers especially), use a headset as a courtesy to others, and stay on mute with large groups until you need to speak. But use the video, because it brings context that you can't see with voice only. See how people react when you speak to them. Pro tip: I like to put the video in a smaller window right under the camera when possible, as it approximates "eye contact" a little better.

And for the sake of humanity, don't get on a call with a dozen people (already too large a meeting) and constantly ask who joined every time the chime happens. Nobody cares. They're late, get your business on.

Be on time, make sure that your equipment works before the start. Every software allows you to preview sound and video. Make sure sound in particular is the right device, like the headset instead of the microphone inside the laptop.

If you use any instant messaging service, make sure it isn't putting up notifications when you're sharing your screen. I've seen the embarrassing notification come up, "Joe has body odor and doesn't know what he's talking about." You don't need Joe to see that.

Use a big group video meeting to have lunch. I know, it's kind of gross to watch people eat, but I can tell you that distributed teams get to know each other better in these social situations. No work talk. 

Be sensitive to interruptions, or interrupting others. Depending on your job, some people need "maker time," and being interrupted is costly. Software developers can need as much as 20 minutes to get into a groove and be productive, and you don't want to stop that. Mute your own notifications, close Facebook, leave others alone if it can wait. If you're a manager, don't ever think that what you need to do is more important and worth the interruption of your makers.

That's my brain dump. Again, you might be working remotely for the first time under some extraordinary circumstances, so this framework and advice doesn't really take that into account. Good luck, and take care of yourself and your family!


Hot dogs

posted by Jeff | Saturday, March 14, 2020, 3:45 PM | comments: 0

Writing helps pass the time, and already the most mundane things are creeping into my head. So I'm going to write about hot dogs.

Fat me, circa early 2005, ate a lot of red meat. By a lot, I mean it was every meal. It wasn't filet mignon either, more like Whopper Jr.'s and that tasteless $1.49 steak at the end of the meat case. To be really gross, a decade before that, I would go on my lunch break from CompUSA to the Marc's grocery store next door for a "death dog," as we called them, because they were the lowest form of questionable meat you could find, and only 25 cents.

That year was also a year of great stress and turmoil for me, as my first wife and I split, I felt professionally wandering after finishing my book, and hey, look at that, my health sucks. I was frequently clocking in at 230 pounds (while 5'8") and walking up stairs caused me to be out of breath. All of the change and uncertainty was a shock to the system, and I responded by trying to control whatever I could. As one sometimes does, I realized that I can control my own body, so I adopted a regimen of exercise, diet change and body piercing. I'll write about the last part another day.

I was making some serious bank doing consulting jobs, and being single-ish with no children, there was a real lifestyle benefit to that. I had gotten ahead financially enough by early summer, when one of my former volleyball kids suggested that I coach her previous high school team. In retrospect, I'm not sure what I was thinking, adding something emotionally exhausting like that to my already emotionally exhausting life, but I agreed, and decided to make that my full-time pursuit for a few months. Being a participating coach, I did the laps and stretching every day, and I needed that routine.

The outcome is that I dropped 30 pounds, and emotionally exhausting as it was, it stands as one of the most meaningful jobs I ever had. This was associated with some changes in eating that I started some time before, including the complete removal of red meat from my diet. I didn't have anything against it, per se, only that it was clearly the source of most of my cholesterol, which was all scary in terms of numbers. Even relative to the volume of fried food I would eat, which was infrequent, the red meat intake was excessive. So I stopped. I haven't had a beef hamburger in 15 years.

A few months ago, the chef at work assembled what I would describe as a picnic food lunch, and knowing about the diverse dietary considerations of people, he did turkey hot dogs. I knew these were a thing, certainly, and I have turkey burgers whenever I can, but even when I'm shopping and buying hot dogs for my kid, one of the three things he'll reliably eat, I pass right over the turkey dogs. Last week I resolved this, though admittedly I had to get over the "mechanically separated turkey" in the ingredients list (they didn't have Ball Park, which just uses white meat).

What a strange sensation it is to eat a food like that after not having it in such a long time. There's a texture thing going on there that I can't explain. The taste isn't the most critical thing, because let's be honest, mostly what you taste is salt. Maybe it's that you don't normally eat other things in a bun the same way.

I had hot dogs.


Empathy, our social contracts and our world perspective

posted by Jeff | Friday, March 13, 2020, 12:15 AM | comments: 0

I've been talking a bit with people lately about my love for Armchair Expert, the podcast that Dax Shepard does with his adorable "soulmate" Monica Padman. (Full disclosure, I've always had a thing for Indian women, and redheads, but genetically, these two things do not mix in real life. EDIT: I was wrong about this.) If you don't know Dax from Parenthood or Baby Mama, he's the husband of Kristen Bell, who is Veronica Fucking Mars. Dax is, as it turns out, an extraordinary interviewer and amateur anthropologist. Monica and her friend Jess have created a sub-series in the podcast called Monica and Jess Love Boys, and it too is fascinating. Jess is gay, 40-something, and has sex with a lot of guys, and Monica is 30 and hopelessly single. Their sub-series is all about the reasons they're single for polar-opposite reasons. It's fascinating. Seriously, listen to it.

In the third episode of this sub-series, Kristen Bell calls the pair out on some of their self-inflicted limitations. She does so from the angle of having worked through her own issues, and to say that Dax and Kristen are the best kind of human beings on the planet doesn't quite capture it. She points out a few things about her own world view. First, she indicates that if Dax were ever to cheat on her, she wouldn't personally believe that it was about her as much as she would try to understand what it is about his life that would cause him to seek that relationship. Her generosity in this sense is astonishing. Second, she says that a lot of the social contracts we adhere to are societal in nature, including the thing about cheating, and that we shouldn't automatically try to make them about us. She gave the example of how Dax has not simply cast aside his previous relationship of nine years and catches up with his ex at least annually. That one I really identified with, because I've had really just four "significant" relationships in my life, and Diana has met them all three (and vacationed with two). I love these women as essential parts of my life, even if it's not romantic in nature. That Bell can be that generous and empathetic deserves many hat tips.

My point is not that I'm looking for an excuse to cheat or anything like that, but rather that Kristen Bell exhibits extraordinary empathy. She removes her own ego and value system to do her best to understand other people. I can't do that the way she does. I want to, but I know I fail at it much of the time. Humans have infinitely different experiences, and they shape who we are, for better or worse. Isn't the first step in getting along to accept that?

In recent weeks, I've tried to understand this in terms of our politics. Sure, some people are in fact horrible racist people, but it's not fair to believe that every Trump supporter is racist, even if they support a racist. Watching the Democratic primaries, the passion for the more progressive candidates is often driven by circumstance, even if those candidates don't have the broadest appeal. I'm finally getting that.

The coronovirus outbreak has certainly brought this to a new level. Statistically speaking, it's reasonably likely that my family will get it and be fine. It's also going to have dramatic effects on communities that I care about, including the Broadway/theater community, and where I live, given the catastrophic impact on service economies (I live in Orlando, so you can do that math). I'm also watching my family and friends in Seattle, the worst place to be in the US at the moment, see their lives upended. (EDIT: At least Disney and NBCUniversal will pay their people through the closures.)

Here at Puzzoni World Headquarters, 2020 was already pretty much a shit show, for a half-dozen reasons I don't need to share online. Like any human, we all have our shit to deal with. In the last 24 hours I've had to cancel a trip and explain to a kid with ASD why we had to change plans, and you can imagine how that went. I wish that was it.

So in the mean time, you bet I'm going to make jokes about my 401k bottoming out, my liquor stash, being holed up with a 10-year-old on spring break, and whatever else crosses my mind. You know why? Because I don't know what else to do. I'm spent. I'm not interested in having some pissing contest about how hard my life is relative to anyone else's, because I've had years of therapy to know that keeping score is bullshit. Our reality is our own.

Last fall, Obama made this great comment about how people needed to stop trying to out-woke each other out of this sense of righteous activism. At the time, I thought he was being a little harsh, but he was right. We're not going to change shit if we keep trying to make a point at the expense of others.


Parenting update: It's gonna be harder before it gets easier

posted by Jeff | Tuesday, March 10, 2020, 11:25 PM | comments: 0

A few weeks ago I checked in with regard to our parenting challenges, and general exhaustion. Things aren't really getting better, but I'm settling into some acceptance of that reality. That in turn, I hope, is going to make me better equipped to roll with it.

Shortly after Simon started on the Abilify, the doctor doubled the dose since we didn't see any appreciable change in behavior. Around this time, we started to see a general lack of interest in eating and he dropped about 5% of his body weight. This was weird because that particular drug is notorious for weight gain as a side effect. In general, I felt that he was lethargic. He had no interest in riding bikes and just wanted to sit around and play video games. I think at most he spent about six weeks on the drug, but seeing no constructive changes, I felt strongly that we should ramp him off of the drug. Additionally, we were seeing a lot of extraordinary anger about, well, everything. We even captured it on video for the doctor to see.

He's been off of it now for about a week and a half, and for more than two weeks, we're having the anger about 80% of the time in the evenings, and full on ASD meltdowns about 20% of the time. I'm surprised that no one has called the police with all of the screaming. My hope was that we could establish some kind of behavioral baseline after the drug worked its way out, but we introduced another variable that makes that difficult.

Some of the anger comes from the competition that video games play against everything else, including homework, but also simple routine things like bedtime. We really allowed him to spend unlimited time with games over the holidays, and I think that came back to bite us. As we explore the accountability vs. accommodation that I mentioned previously, these basic contracts are difficult for him to accept. In any case, we now have a small base allowance for game time, using a timer, and additional time is granted based on homework and reading time. It all goes off at 7:30, without fail.

The time trade started this week, so we don't have a behavior baseline because we changed the rules again. And dammit, next week is spring break, so we're back into the same trap we had during the holidays. There is one sliver of hope though... Simon is self-aware about the hair-trigger anger. Once he's calm, he feels genuinely bad about, and doesn't understand why he can't control it. These anger fits are pushing us to the limits, but once we get into a rhythm with the earned game time and limits, my hope is that we can then identify with confidence what his "normal" is under these rules, and take that to his doctor for advice.

For me personally, I'm getting better at compartmentalizing the chaos and the sadness that he feels. It hurts to see him like this, but just as I know he doesn't have the skills to cope with what he's feeling, I can't let my own frustration and sadness take over. Sure, that means I have to find ways to vent and exercise self-care, but I'm working hard to figure that out. Diana is being an all-star, but it's taking a toll for sure. We tag out a lot, but it feels like she takes the worst of it, because she's better at helping with homework than I am. Simon prefers her over me in most situations, so that puts more of a burden on her.

I'm optimistic that we'll figure this out, because I don't know how else to be.


There is too much content (and not enough real news)

posted by Jeff | Wednesday, March 4, 2020, 11:00 PM | comments: 0

As someone trained to produce media, with a journalism degree, I saw a couple of things today that triggered me a little. The first was some random person in one of my forums crediting "YouTubers and podcasters" as the source of information that they believed was true. The second was a Disney fan site reporting the "news" that restaurants are no longer serving bread with dinner, followed by countless comments about how Disney is greedy and destined to fail. So many groans.

When the Iraq war started in 2003, we had our first glimpse of having too much time to fill. It was the start of the decline of really solid international news coverage. At the time, CNN had its sister network, Headline News, which repeated the news most of the day on a 15 minute cycle. As things happened and became more newsworthy, they rotated out stories for the newer items. I loved that format, because you could get a lot of information with little commitment. But Fox News in particular put up their waving flag graphics at the start of the war and did their best to fill 24 hours of talking heads, every day. MSNBC eventually followed but was irrelevant for many years. By 2013, Pew found that the cable news networks were reporting news less than half of the time, and MSNBC was the worst with an 85/15 opinion to news ratio. It didn't take long for Headline News to be a format that was not profitable.

Many people, including me, suggested that big news organizations had lost their way, incapable of separating opinion from fact. I think this is objectively true. There has always been a more subtle debate about whether this came about out of profit motives or just giving people what they want, and I think that's objectively both. The problem is that it didn't end there. The great promise of the Internet was that it gave everyone a voice, but people seem to be indiscriminate about the voices they listen to. That's why we have people like the ignorant "Food Babe" able to make a living ("did you know they pump nitrogen into airplanes?!").

I'm not even going to get into all of the crazy racist stuff your uncle posts on Facebook.

When I was a kid, and really until he stopped reporting in 2005 because of cancer, I watched Peter Jennings on ABC News almost every night. I strangely associate 9/11 with him, because he was on the air for 17 straight hours. Jennings, along with Tom Brokaw and Dan Rather, both of which retired around the same time, were the classic news men of a different era. We could trust them and expected them to give us straight facts. They weren't perfect, but they led organizations that earned our trust.

Now, even the people who remember those journalists, have largely abandoned truth and pursued the voices that will make them feel comfortable. That's sad.

As for the bread at Walt Disney World, maybe you haven't noticed, but bread has fallen out of favor because it's a carb and people want to avoid all of the gluten that wasn't hurting them. Maybe they stopped serving it just because it wasn't what customers wanted.


Simon, 10

posted by Jeff | Wednesday, March 4, 2020, 3:20 PM | comments: 0

My darling boy turns 10 tomorrow, which seems even more impossible than 9. Not much has changed since last year, which is to say that he struggles in many of the same ways, while successfully growing up in other ways.

There were some solid wins this year. He continues to improve in swimming, and after having a bike for two years, he finally learned to ride it. He independently rides to the bus stop every day. He exhibits a ton of independence in certain contexts, including on cruises. We give him a cheap phone with the cruise app, and he roams about the ship doing whatever he wants, checking in with us (it's not like he can disembark!). He has, on and off, engaged with kids in the neighborhood, and tried to have sleepovers. He's seeing a therapist regularly and seems to be comfortable talking to her. He's getting better at articulating what he thinks and feels.

Our struggles this year are again with school work, and again trying to find the right drug fit for ADHD. We're coming off of a really bad match right now that caused weight loss and erratic moods, so I have to be careful to characterize that as the last year in its entirety. He has an amazing doctor who is helping us find the balance between accommodation and accountability, but this is super hard because it's contextual to every specific situation and hard to broadly categorize. The classic challenges around social contracts and ASD will be with us for years to come, so we just have to keep looking for ways to work around those. I just wish he could find the one friend who can roll with who he is.

We have some good laughs most every day, and I'm so grateful to have this little human in my life. We're celebrating 10 this week, so here's a look back at the year, in pictures.

This one sticks with me. We witnessed the ultimate anxiety driven meltdown last April when the FSA testing began. His previous school fostered a toxic obsession with testing that no kid should have to endure. Fortunately, the maps were redrawn and he was switched to yet another new school this year, with a sane principal who advocates for him.

I brought Simon to work, which was a lot of fun (even if I didn't get anything done that day). It's hard to explain what your job is when you lead a bunch of software engineers, but he thought it was cool no matter what.

We started to consistently give Simon a few bucks per month, and I think it was the first time that he legitimately bought something. A neighbor was selling this truck at a yard sale, and it came home with him. Subsequent purchases have been for downloadable content on video games, which I really don't want him to spend money on, but I guess he has to learn about value and saving for himself.

I'll never forget the first time we cruised, about seven years ago, and he didn't even want to stand barefoot in the sand, let alone get in the water. Now the kid is swimming really well, with short and focused weekly lessons. He gets out into ocean like a boss now. I keep hoping that he'll consider swimming competitively, but he has to find the confidence for it.

Simon missed the entire first week of school being sick. I felt like it was a bad omen, given the struggles he has had with completing work.

He got to hold his first baby this year, Selene, my best friend's first child. He was fairly enamored with her, though she was content to sleep.

The Millennium Falcon landed at Galaxy's Edge this year. I am so amazed at the way that Star Wars has crossed so many generations. We're pretty lucky to have this in our back yard.

We did add a few new rides to Simon's experience this year, most of them at Universal Orlando. We still haven't been able to get him on Rock-n-Rollercoaster, the last ride at the Disney parks that he hasn't been on.

Castaway Cay is my happy place, and part of the reason for that is the happy times we inevitably have together there. We like to get out there as early as possible and stay until the crowd starts to thin out. As a confident swimmer, he's happy out there.

The Skyliner transportation system opened at Walt Disney World this year, and it's a wonderfully calming and fun thing to do, even if the parks themselves are crowded.


Still glad I built that PC

posted by Jeff | Tuesday, March 3, 2020, 4:10 PM | comments: 0

Back in July, with bonus money burning a hole in my pocket, I built my first PC in more than 14 years. I did follow up that build with a second monitor and a stand for both, since the original stands were like a Jell-O® mold wobbling when I typed. So my all-in cost was actually $2,100 in the long run. Again, not cheap, but still way less than the single-screen iMac I had bought four years before. It was a great purchase.

Having two 27" 4K monitors is life changing. It's staggering how quickly the prices came down for more pixels. I'm a huge fan of painted-on text. For the longest time, I was pretty satisfied with a single 27" monitor, I suppose because it felt gigantic relative to the days of having 20" or 24". When I'm doing actual work, I tend to not have anything open that isn't necessary, so Alt-Tab-ing around isn't a big deal. But the biggest change in the last few years, when you're writing code, is that you've got all of these distributed "things" doing work all at the same time. While it isn't strictly necessary, it gives me a certain amount of comfort when I can see that something in a browser happens which makes something in some serverless thing happen over there which lights up something else. Sure, we write all kinds of tests and automation to make sure all of this loosely coupled stuff works together, but particularly early on when you're prototyping and such, it's nice to prove that what you're doing actually works.

Speaking of all that compute stuff spread around, lots of cores and memory is cheap and easy these days. Eight cores and 32 gigs of RAM is overkill for everything that I'm doing. For example, to run the hosted forums locally, I have two web apps, a bunch of Azure functions, SQL Server, some queues and both Redis and ElasticSearch running in containers. Then with Visual Studio running, plus VS Code (where I'm writing the documentation), a bunch of Chrome tabs, music playing... I still don't think I've exceeded 18 gigs of RAM actually in use. My CPU cooler changes color when it gets hot, and it's always green.

I'm still a little nervous about everything being straight-up Windows, as opposed to the Mac with Windows VM's that I can reset or throw away, but so far I haven't experienced any OS rot. The only thing I really miss is having the last non-subscription version of the Adobe CS apps, but honestly, they were barely working on the newer Macs, especially since they all pre-dated high-resolution displays. I am buying the Photographer bundle with Photoshop and Lightroom, but I miss Illustrator, Premier Pro and After Effects, even though I didn't use them that frequently.

One final note about the build, it would be quiet if it weren't for the power supply. It's not like a jet engine, but it makes some minor white noise. It's unfortunate, because the liquid cooler fans and case fans are nearly silent. It's not a deal breaker, and to be fair, my previous iMac did sound like a jet engine every time you did a long running build.

Maybe the best part of building my own thing is that its shelf life is fluid. When you can replace parts, it's not just something that you try to get three years out of and replace entirely. As long as the various interfaces stick around for awhile, the innards will connect to each other. I haven't paid attention enough to know if it's all just bus speeds and sockets changing. Certainly the case, the SSD's, video card and the cooling hardware should last for a bit. The CPU, RAM and motherboard might need to be upgraded together, but I'm not starting from zero regardless.


POP Forums, the product launch retrospective

posted by Jeff | Sunday, March 1, 2020, 12:30 AM | comments: 0

At some point over the last six weeks and change, I launched POP Forums as a commercial, hosted product. By that I mean, it was something I could sell to others. I'm not suggesting that I've done any real sales effort, obvs, because I don't have any external customer yet. But there was a lot to learn from the effort.

The first bit was that, in instrumenting the shit out of it, I discovered some weaknesses early on. Maybe "weakness" is a strong word, because none of the things I found would have prevented a new, high scale customer from bringing it down. I had, over the course of the last year and a half, written all kinds of code to make it scale, but never really put the app into a real scale situation, where there were multiple nodes and an external service (ElasticSearch) providing search, etc. There were some quick lessons learned:

  • Having search well outside of the network where the app runs means it did sometimes fail. Retry logic (using Polly) adequately addressed the problem.
  • At first I tried using the AWS flavor of ElasticSearch, but I was kind of annoyed at their overall approach to security, which was different from that of straight ES, meaning it was like all permissions in AWS. There were way too many cautionary tales about the speed at which you could scale the AWS flavor, too, and starting out tiny, that didn't sit well with me.
  • Moving my search to an Elastic hosted instance was the right move, and it's dirt cheap for the foreseeable future. Like, under $20 per month cheap. Plus it's "real" ES. The only negative is that it's hosted in Azure US West-2, and my app is in US West. Search results still come back in a decent 150ms-ish hit. It's easy and fast to scale up or down.
  • The small utility app I wrote to populate the search index took about three minutes to run for a half-million posts. Again, the cheap index hosting from Elastic took it like a champ.
  • The Linux variety of Azure App Services are actually wrapped up in Docker containers, even when you deploy via web deploy. This might explain why they deploy and scale out so fast.
  • The forum app and the management app are both running on the same app service plan (the same VM), and I'm running on two instances for now. I had one of the management apps go "bad" for some reason twice, so depending on which instance the infrastructure assigned your request to, it could have returned a 502 bad gateway response. Not sure why Azure doesn't automatically sense this as a failure and take the instance out of rotation, but they have a preview feature for this. I'll take it!
  • I have to keep the ARR affinity on, which means users are sticky to a particular node, otherwise SignalR, the websockets implementation, doesn't work. This doesn't seem to have any material impact on performance.
  • Web deploy to Linux app services isn't "done" when the deployment is done. It appears that there are some switches in the background as it cycles between old and new instances, and it is possible to briefly get a 502. Maybe having 3 nodes would improve this, but it's a little disappointing, however momentary.

All of this still leaves me in awe that I can have this kind of redundancy and resiliency for so little money. How did we get along before "the cloud?" I've learned that more than 90% of my database calls are handled by in-memory cache (across multiple nodes, no less). The database DTU (the virtualized unit of CPU for the price structure I'm using) averages less than 4%, while the CPU for the web nodes stays well under 10% at the current, starter pricing levels. Page render times stay below 40ms, and that's accounting for serving images out of the database. I'm still confident that I could hit 2,500 request per second without scaling up (though response times would be better if I did).

The "big" test came when I migrated the PointBuzz forums over to the platform, and wow did it shine. It's not a huge load, but 30 to 50 requests per second isn't unusual for that site, and it doesn't cause any deviation from the above numbers. I'm so happy to split the forums from the rest of the site, so they aren't dependent on each other. The search doesn't get a lot of use there, but it's so nice to have real, useful search! The migration also forced me to add a feature to insert Google AdSense ad slots into the page, because their "auto ad" thing wasn't really inserting anything. I added code to insert a banner at the top, and a sidebar ad, and wouldn't you know it, it's generating CPM's of almost $2. Haven't seen money like that in a long time!

At this point, I just need some real customers to prove this is all for real. Honestly, it all works better than I expected.


Awesome custom metrics using Azure Application Insights

posted by Jeff | Sunday, February 23, 2020, 7:35 PM | comments: 0

You know that old management saying, that you can't improve what you don't measure? It's annoying because being measured feels personal and uncomfortable, but it's useful in technology because it's totally true. There is a fairly lucrative market around monitoring stuff in the cloud now, because as our systems become more distributed and less monolithic, we do less tracing at the code level and more digestion of interaction data between components.

When I launched the hosted forums, the big change was that I was finally exercising the two-level cache code that I wrote for POP Forums a long time ago. It was of course inspired by the way StackOverflow handles their caching. Basically, the web node checks to see if it has the object in its local memory, and if not, it goes to Redis. If it's not there either, then it goes to the database and puts it in Redis. Then, using Redis' pub/sub messaging, all of the web nodes get a copy and it holds on to it there with the same TTL. The default that I use is 90 seconds, but I don't know if that's "good" or not. I had no context or data to support that it worked at all, actually.

The first step for me was to plug in Azure Application Insights, because it's silly-easy to use. You add a NuGet package to your web app, a little Javascript to your pages, the app key to your configuration, and enjoy that sweet sweet stream of data in the Azure portal. Instantly I had this beautiful map of curvy lines going between databases and web apps and Azure functions and queues and... not Redis. OK, no problem, a little google-foo reveals that the SDK doesn't have the hooks to monitor Redis calls. So I wander through the Azure portal to my Redis instance, and look up the numbers there. What I find is that there are almost three times as many cache misses than hits, which implies that data isn't repeatedly fetched very often, or something else.

This is actually not super useful, because in theory, my two-level cache would not need to go all the way to Redis if the object is locally stored in the web node's memory. It's also not useful because I wanna see all of this in the pretty application maps!

OK, so with the underlying forum project being open source, I didn't want to bake in this instrumentation because I don't know what others are using. Not forcing them to use App Insights, I decided to wrap all of my cache calls in a simple start/stop interface, and wire in a default implementation that doesn't do anything. That way, a consumer can write their own and swap out the dependencies when they compose their DI container. For example, here's the code wrapping a write to the memory cache:

_cacheTelemetry.Start();
_cache.Set(key, value, options);
_cacheTelemetry.End(CacheTelemetryNames.SetMemory, key);

The _cache is an instance of IMemoryCache, and _cacheTelemetry is an instance of the interface I described above. One could argue that there are better ways to do this, and they're probably right. Ideally, since we're talking about something event based, having some kind of broader event based architecture would be neat. But that's how frameworks are born, and we're trying to be simple, so the default ICacheTelemetry is an implementation that does nothing.

In the hosted forum app, I use a fairly straightforward implementation. Let's take a look and then I'll explain how it works.

	public class WebCacheTelemetry : ICacheTelemetry
	{
		private readonly TelemetryClient _telemetryClient;
		private Stopwatch _stopwatch;

		public WebCacheTelemetry(TelemetryClient telemetryClient)
		{
			_telemetryClient = telemetryClient;
		}

		public void Start()
		{
			_stopwatch = new Stopwatch();
			_stopwatch.Start();
		}

		public void End(string eventName, string key)
		{
			_stopwatch.Stop();
			var dependencyTelemetry = new DependencyTelemetry();
			dependencyTelemetry.Name = "CacheOp-" + eventName;
			dependencyTelemetry.Properties.Add("Key", key);
			dependencyTelemetry.Duration = new TimeSpan(_stopwatch.ElapsedTicks);
			dependencyTelemetry.Type = "CacheOp";
			_telemetryClient.TrackDependency(dependencyTelemetry);
		}
	}

When you add Application Insights to your web project, you add services.AddApplicationInsightsTelemetry() to your Startup. Under the hood, this adds a shared instance of TelemetryClient to your DI container. This is what we use to pipe data back to Insights.

DANGER! A lot of the code samples in the SDK documentation show them creating new instances of the telemetry client with default configuration. This is naturally what I did first because I didn't read the documentation in any particular order. Something about this is bad, because my memory usage spiraled out of control until the app stopped responding. My theory is that it didn't have the right config, so it bottled up all of the events in memory and never flushed them, but I don't know for sure. Meh, using dependency injection is always better for testing anyway, so do that.

As shown earlier, the cache calls are wrapped in start and end calls, so all we're doing here is starting a timer, then stopping it and recording the cache event as a DependencyTelemetry object. For extra flavor, I'm recording the cache key as well, so if something is particularly slow, I can at least infer what the cached data was. For example, a key like "pointbuzz:PopForums.PostPages.17747" tells me that the tenant is "pointbuzz," and the rest says it's a bunch of posts from the topic with ID 17747.

I'm also giving the event a name that's prefixed with "CacheOp-" and not just the operation. Why? Because the lovely map view will group these together when they start with similar names. I learned this quite accidentally, because all of the database calls to tenant databases have a prefix on the database name. That worked out great because it groups the calls to tenant databases from the master database that describes the tenants.

Let's see some raw data! We can enter our Azure Insights in the portal, go to Logs, and query for the dependencies we said we would send:

 

There's a ton of context in every one of these entries, because they all roll up to the request data from the user. So yeah, I find a lot of weird cache misses from Russian bots that I assume are trying to spam forum topics that are 10 years old.

So that's cool, but remember that my original intent was understanding what the hit ratios were for the cache, for both in-memory and Redis. Well, we can write a query for that as well, and even pin it to a dashboard, if we so choose:

Behold! Useful data! This makes way more sense than my initial look at Redis data. It shows exactly what I would hope for, in fact: There are a ton of cache hits to the in-memory cache, and when those miss, they're persisted to memory and Redis. The bottom line is that the juicy data is right there in the memory most of the time. That's success! If you go back to the raw data above, you can see that those memory hits have a duration of zero, so less than a millisecond. That's obviously faster than crossing the wire to the database.

You can also get a good summary of what's going on, and track down slow hits, by going to the application map:

Right away, I wondered what the deal was with the slow gets from Redis. I'm still looking into it, but it looks like there's some overhead and warm up involved with the connection multiplexer in the StackExchange client. It's used so infrequently that there often isn't a connection to Redis established. That may partially explain why the SetRedis events are so short, because there had to be a connection to find there was nothing there before writing to it. It's all speculation at this point though, and generally pages are arriving in under 100ms.

Azure Application Insights is a very cool tool, and not limited to just the basics. It's important that your monitoring goes deeper, because the lesson I've learned time after time is that contextual monitoring is the thing that will save you. Nothing knows about the health of the system better than the system itself, so you should strive to have it tell you what's up (or down, as the case may be).


Technical validation at last!

posted by Jeff | Friday, February 21, 2020, 11:05 PM | comments: 0

Tuesday night, I migrated the PointBuzz forums to the hosted forum app, and the site that served as v1 more than 20 years ago came full circle as the first customer. Both of the sites had been running on a similar version for some time, but there's something different going on here that provides a mountain of technical validation that I didn't have before.

Without nerding too hard, I really wanted to prove that all of the stuff I wrote to scale the app worked. Not worked to scale, but worked at all. If you go back in history, the forums were always written to be this self-contained thing that would work on shared hosting (hello 1999!), so breaking it up into cloud-first pieces was a lot of work. But several big things worked out:

  • I moved search to ElasticSearch. In my case, I'm using the managed version from Elastic itself, running in Azure. I'm spending less than $20/month to start, and while it's not super fast at that spend, it gets real results that are useful! My home-grown solution wasn't terrible, but it wasn't great.
  • All of the background stuff that happens like sending email and indexing text for search, happens in serverless instances (Azure Functions). What's awesome about this is that it costs a few bucks for millions of executions.
  • It all works across multiple nodes! Since moving to the cloud, I always wanted to have more than one web head just for redundancy. There have only been a few times where I'm sure the node failed and a new one was spun up, but having two is nice. I have to keep the affinity on (sticky sessions) though in order to make SignalR work, but this doesn't seem to really impact performance.
  • The two-level caching works. I use StackOverflow's pattern here, sort of, replicating and invalidating the cache in local memory by way of the message bus in Redis. I don't have a great way to measure the hit/miss ratio in local memory, but I imagine it's pretty great. User data is what gets cached the most. I'm estimating that with PointBuzz alone, I'm avoiding the database about 50,000 times per day. That gets really important with more customers. (Sidebar: This failed a lot when I was prototyping a few years ago, as the old version of the StackExchange Redis client was prone to failure.)
  • Average response time went from about 30ms running on a Windows app service, to 5ms running on a Linux app service with about the same "hardware" and the same database. That's nuts! It's the same code! Maybe there's something I'm missing, but the only other thing that I can think of is that it's now running multi-tenant, which adds overhead, not reduces it.

I'm also trying out Azure Insights for monitoring. It's pretty cool, but the cost seems a little steep so far, I think averaging a buck a day for relatively low traffic. It also doesn't monitor the Redis service. Like any service, you can feed it custom events, so I might be doing that to instrument the caching both local and on Redis.

So now I just need customers. Selling is not my strong point, but I have some ideas to get the name out there and perhaps line up some "free" customers as promotion. I have a reasonably baked product, and that's very exciting.


Parenting status: exhausted

posted by Jeff | Thursday, February 20, 2020, 9:30 PM | comments: 0

I haven't written about our parenting adventures in a long time, in part because it's really hard to narrow it down to any particular category of action. Looking at it in the broadest of terms, we have this kid being a typical almost-10-year-old while rolling with aspects of ASD, ADHD and anxiety. It's a lot to process, and there are days where I just wish he could be a carefree kid that things come easy to. That certainly wasn't me as a child, so I'm not entirely sure why I would expect it to be true of him.

School this year is sometimes a struggle, though it's not as bad as it was last year. Without having a Type-A overachieving principal (I need to publish her emails about testing that I FOIA'd from the district some day) this year, Simon's homework load is lower, and they're not beating the kids up about standardized testing. He does get some homework, and it's a consistent struggle to get it done. This is where our team and the collective advice was not serving us. Sometimes for ADHD, you need to chunk-up work, which is a logical strategy. However, we were noticing that the ensuing freak-outs about getting it done could be easily diverted, the "squirrel!" moment (familiar if you're familiar with the movie Up). His developmental pediatrician, the one prescribing the drugs, immediately called bullshit on this as typical age-appropriate behavior, not ADHD. He's been working the system so he could do the things that he wants to do.

However, throwing this back to his therapist (which is not at all covered by insurance, we recently learned), she was a little taken aback by this. It puts us into a weird accommodation vs. accountability problem. Sometimes he does need to be granted accommodations, but not always. There's no magic formula for this, because it's completely contextual in the moment. It makes it harder for us and his teachers, who are bound by his IEP.

Then there's the video game problem. Let me first say that I can't in good conscience write off games as "bad." I loved them, and computers, as a kid, and this love was almost always treated as an annoyance or waste of time. I won't do that to Simon. I'm even more cautious now, because I'm finally seeing some level of creativity when he builds things in Minecraft. This was my concern about his Planet Coaster obsession, where he mostly downloaded other people's work, but never made anything. I believe creativity is one of the biggest contributing factors to success, and I want him to exercise these new muscles.

Now the issue is that it's one of the things he most cares about, and it can be at odds with doing homework. A funny thing happened when we said he couldn't play games after school until his homework was done, but he could have a necessary break before doing it. The homework got done in bigger chunks, a little faster. It isn't consistent yet, but it seems to be working. We'll see how that plays out. At the very least, it makes some of the learning challenges a little more apparent. For example, reading comprehension isn't really achieved by scanning for keywords, and he clearly does that. Knowing that, at least we can help.

We've asked the therapist to focus more on social skills and coping mechanisms for difficult social interaction. Simon's social struggles are fairly typical autism things, where he fundamentally doesn't get social contracts. This got him into potentially serious trouble at school when he joked about something inappropriate. You can imagine how that rolls for a kid who is under the impression that being funny is what makes people like you. Then throw in the random thing where kids say they aren't your friend for no particular reason, and you know how that goes. He desperately wants to have close friends, but at the same time can engage in really antisocial behavior. He is me, and it took me years (and a lot of therapy) to figure that out.

We're also in the midst of drug switching, again. The ADHD meds seem to be working OK, but we took him off of the anxiety drug because it clearly wasn't doing anything. The doctor replaced it with Abilify, which is used to treat a wide range of more serious conditions, but in his case, it's intended to even out his moods and help keep the anxiety in check. So far, it's hard to say if it's working, but it's definitely making him tired. Fatigue becomes yet another variable to consider, and then you're not sure if we're making progress. I suspect the ADHD drugs will stop working in the next 6 to 12 months too, just because none of them seem to help for a long duration. 

I hate this drug experimentation the most. There are no silver bullets, because everyone is different, but non-scientific me feels like I can't know who Simon really is. I need to get over that, because we know what no meds looks like, and it's a mind that's always racing.

All of that challenging stuff aside, we've certainly had some wins, too. Simon finally figured out how to ride a bike, thanks mostly to Diana's persistence over a few days. We bought him a bike two years ago, and he's finally riding it. In fact, he rides it to the bus stop now, locks it up, and rides it home after school. He is also (or was, before the drug switch) consistently getting himself out of bed in the morning and coming down for breakfast. Given our concern about accommodation that might have been coddling, you can imagine how good it is to see all of this personal responsibility.

While he doesn't generally fit in with many of the neighborhood kids, there are a couple that he seems to connect with at least some of the time. His interests are often about video games, so that's where the connection often is. It's difficult to teach him kindness when little is afforded to him, but we see it now and then. It's a work in progress.

We're a little exhausted. Parenting is not without joy, but I never imagined it would be this hard. Simon is a smart little boy, a little different at times, but I have to believe that someday it will work to his advantage. I find myself worrying less about his long-term outcomes lately, and more about him just having happy times as a child. When he's 30, I want him to believe that we did right by him, to the best of our ability.


Documentaries and Cheer

posted by Jeff | Saturday, February 15, 2020, 11:15 PM | comments: 0

Diana and I recently finished watching the 6-hour-ish documentary series Cheer on Netflix (or "docuseries," as apparently everyone involved in it is contractually obligated to say), and it was amazing. I thought while stream-surfing that maybe it was a reality show, but no, it was a bona fide documentary, and it was great. I went to a high school that actually took cheerleading seriously as a competitive sport, even if it wasn't in a local context, so I've always looked at it as a legitimate athletic endeavor. The documentary only reinforces that stance. I ended the series feeling like I wanted to adopt every single one of those kids and hug them and tell them that they're awesome. That's the mark of an effective documentary.

I've always had a thing for documentaries, and it somewhat relates back to the concept of "gonzo journalism" that Hunter Thomas brought about back in the day. The idea that you could observe something while influencing and participating in it, and still tell a story about it. For documentary film and television, this came about arguably with the start of MTV's The Real World. That's when they said, "This is the true story, of seven strangers, picked to live in a house, work together, and have their lives taped. Find out what happens, when people stop being polite, and start getting real." This was technically a documentary, but it was also a contrived and made up reality. Of course things get uncomfortable when you force a racist and a black activist to live together. There's nothing about it being "real" when you force it.

Documentary in its more pure form observes a situation that would have happened whether there were cameras there or not, and it's infinitely more satisfying, if somewhat voyeuristic, to see. Cheer checked all of those boxes. Our personal relation to it was that we stayed at a Daytona hotel next door to the band shell where the cheerleading championship took place. (For the record, Daytona, and those hotels, are a total shit hole.) We were there possibly around the time of the Cheer doc. I can't stand things like Survivor or The Bachelor, but Cheer was legit, because it wasn't contrived for TV.

I wish there was more of this on TV/streaming services. Unless you're a robot, documentaries are an instrument of empathy creation for your fellow humans. I think that's valuable. It's also the kind of thing that I aspire to. In high school I followed around a cheerleader during tryouts to show what that process was like, and that aired on cable access. In my first real job, I followed around high school freshman to show what it was like to start high school in an over-crowded school. I did a mini-doc after the fact to talk about the creation of a roller coaster. It's my thing even if I haven't pursued it on purpose.

The upside of the streaming-ization of TV is that there's more of this available than there used to be. That's exciting. I love documentaries.


The American Adventure

posted by Jeff | Saturday, February 15, 2020, 6:50 PM | comments: 0

If you've been to Epcot at Walt Disney World, you've probably seen The American Adventure, the audio-animatronic show that chronicles the history of the United States. The thing that I love about that attraction is that it's a surprisingly brutally honest review of our history. It doesn't gloss over slavery, the Civil War, the destruction of Native Americans or the wanton destruction of the environment. Despite all of the negatives, somehow the great experiment persists.

I always leave that attraction simultaneously proud, sad and hopeful. There's a montage at the end, to the original song "Golden Dream," where they show so much of our achievement, including shots of Space Shuttle astronauts, Dr. King, Kennedy's "Ask not" speech, famous Olympic athletes, Steve Jobs, Peter Jennings, etc. Our history is legitimately dark, but we've slowly been able to overcome some of it. It makes me want to be a part of the generation that helps us move above our worst.

I was a product of the tail end of the civil rights movement, specifically the part where desegregation was a forced issue in the schools. If environment is a deciding factor in how our views are shaped, I'm thankful for that experience. It's not that I don't see color because of it, it's that I see that color is not the determining factor in how people are valued. To not see color is to ignore the inequality that persists.

We were watching The Help today, and so at dinner, I had to explain to Simon what racism is since he wandered into the room while it was on. "Those people were stupid," he said, even though we're trying to discourage the use of that word when describing people. If it's really going to take another generation to shake racism out of our culture, hopefully I'm doing my part.

I desperately want that optimistic version of America in that Epcot attraction.