Literature survey · 2024 · with Meiyappan Nagappan
We have spent a lot of energy asking whether AI writes good code. I got curious about the question sitting right next to it, the one almost nobody was measuring: what happens to the human?
So we read everything we could find on it - 88 user studies of real people working with LLMs on real programming tasks - and pulled them apart looking for patterns. We looked at Behaviour. How do programmers actually prompt? Where does their time go? Does the tool genuinely make them better, or just faster, or neither?
What we went looking for.
Four questions: How do programmers interact with LLMs? Does using one actually enhance the human's ability? Does it improve the work itself? And are there specific interaction habits that lead to the gains, versus the ones that quietly cost you?
What surprised me.
The story people expect is "AI makes coding faster." The real story is variability - and it turns out both parties are non-deterministic. The model is unpredictable, sure, but so are we. The same tool produced wildly different results depending on how the person drove it. A few patterns held, though. People spend most of their time not writing but reviewing - crafting the prompt, then trying to understand what came back. The job quietly shifted from writing code to reading it. Time gains were common but not universal, and complex tasks or shaky expertise could flip the tool from help to hindrance.
Why it stuck with me.
The paper's real conclusion is that the interaction is the product. Raw model capability is not what decides whether an LLM helps you - the way you two talk to each other is. That idea is now the spine of how I design AI features: the guardrails and the conversation matter more than the model.
A note on framing: this was my Master's research, co-authored with my supervisor and hosted on arXiv. It is not peer-reviewed. I still think it is worth reading, and I would rather tell you exactly what it is than dress it up.