This is just an anecdote, not a study, but I thought I'd share some recent observations about using AI for coding. While still firmly in the realm of being a great boost for productivity, I am a little worried about what happens with the folks, especially inexperienced developers, that just rubber-stamp whatever it comes up with. This is in the context of mature, larger code bases. The big design and architectural decisions have been made and it's well documented.
First, I ran into a UI bug where a disabled button still did something when clicked, and it was unexpected. There's a state box that says the button is disabled, so one would think that's how it would decide not to act. On the first try, Claude modified a huge block of code that was used to defensively work against a non-related problem (network latency or failure), confidently declaring, "The problem is clear." Wrong, revert those changes. Second try, Claude checked the CSS class on the button with magic strings to decide whether or not to do anything. Also wrong, because certainly the state of the DOM is not the source of truth. I just gave it the line number, and told it, "Check the value of this boolean, and if it's false, don't do anything." That was a huge bug waiting to happen if the CSS ever changed.
On another project, there is a reusable component that appears in three different places. What's different is the source of the data. Still, the AI agent keeps trying to make secondary versions of the component, or try to override and modify it peripherally. Again, the functionality never changes, but I've had to course-correct it over and over again.
Individually, these are not a big deal. However, this is how you get to the "death by a thousand cuts" scenario. And to be clear, this is no different than working with humans. The experienced folks need to guide those who are less experienced, who in turn do that for others down the road.
The argument that I keep hearing is that, "The models will get better." Will they? They're trained on code with the same problems. Because LLM's lack wisdom or judgment, they don't "know" what the best practices are. If human feedback is really being fed back into the machine, I'm seeing more slop, not less. And for those who suggest more rigor and process, haven't we spent literal decades trying to reduce process to get to market faster? Human curation and guidance, by comparison, are cheap, and I don't mind guiding the robots. I don't want to go back to armies of analysts and reams of documents to build something that makes countless incorrect assumptions before getting in front of users. Do you?
No comments yet.