Have you noticed that almost everyone in technology puts "AI" into their title now on LinkedIn? I saw one that said, "CTO, AI native." Not sure what that means exactly. If we're being analogous to other "native" usage, that would mean you don't know life without it, in which case, that CTO would be, what, 5 or 6-years-old? I kid, but all of the chest thumping, anecdotal and hyperbolic claims, and frankly people just selling something (or themselves) make me uneasy and skeptical. It reminds me of "wellness experts" on other socials. I think that a little humility is in order.
So I preface this as saying that this is only opinion, and my anecdote. I'll get to the academic research in a moment.
I believe that once you get the "right" interaction with AI, that it is a useful tool. I find it to be excellent for writing a lot of boring boiler plate code, and implementing stuff with libraries and frameworks that I'm not deeply familiar with. I don't have to wade through documentation or StackOverflow, I just ask it how something should be used. That stuff is downright fun, and feels like magic. I can definitely do more than I could without it, which feels like a productivity gain. Admittedly, some of that gain is just not having to rely on my ADHD brain to stay engaged. I really am having more fun that I have in a long time.
There are (at least) two caveats though. The first is that AI expresses confidence that isn't rooted in wisdom. To prove this point, I asked it about an architectural decision, and it suggested what I would consider an anti-pattern. In response, I intentionally gave it another anti-pattern that I said was better, and it said something like, "You're right, that's a much more [superlative] solution for your scenario." It definitely wasn't. Put that in the hands of a more junior developer. They can be super smart, but at that stage in their career, they don't have the same wisdom as a veteran. They may have a working solution, but anti-patterns don't make performance, security and durability problems go away.
The other thing is that context is ephemeral, if the AI has it at all. For example, I was working with a library to do video transcoding, which is far out of my realm of expertise. We went around with four implementations, none of which would have performed well, or would have been cheap, even though I expressed this as a requirement. But most importantly, it never considered that iPhones make QuickTime files by default, not generic mp4 files. When I pointed this out (with another "you're right" response), it forced an entirely new approach. I didn't know what I didn't know, and neither did the AI.
My point is that while the AI is quite good, it's not as good as what people claim it is. And tech bros always have reasons why that's on you, and not the AI. If it's me, then the tool isn't as good as they suggest it is. That's how my teenager responds when I call him out, he blames his shortcomings on us. While I saw huge improvement last year, it was more subtle this year. Maybe we'll see another leap, but we've also been told that self-driving was just a year away, for ten years.
Academically, it's a solid but mixed picture. The Becker et al. study last year concluded a negative productivity change, and that was a high quality study. Another study noted productivity gains among more junior developers, but another study observed less durable code with more churn. There was a 40% increase in secrets getting into code in another study. The Faros report last summer had the best metric: AI did not improve team velocity or correlate to better business outcomes.
I think folks are so hung up on what AI can do that they're not asking about the intended outcomes. As most non-naive people understand, the business outcomes are the only measure that really matters. If you built an app last weekend and it put you permanently on the beach this weekend, great, you're a unicorn. For now though, I think there should be more energy devoted to looking at outcomes, and working backward from there to make AI truly useful.
No comments yet.