Empirical study · 2023
User research is the part of building software that everyone agrees is essential and nobody has time for. It is slow, it is expensive, and most teams quietly cut it. So I asked a slightly heretical question: could ChatGPT just... do it for us?
I decided to actually test it instead of guessing. My case study was one I knew in my bones - a church choir management system, because I have sung in enough choirs to know exactly how duct-taped those processes are. I interviewed three real choir members about what they needed. Then I fed ChatGPT their profiles, asked it to role-play each of them, and had it generate the requirements it thought those people would want. Finally - the fun part - I took the model's answers back to the real humans and asked them to rate how useful it got.
What I found.
ChatGPT was good. Unnervingly good, actually. It matched most of the themes the real people raised, and the users rated a lot of its suggestions genuinely useful. But it always missed something - at least one category per person that only a real human thought to bring up. The most interesting miss made me coin a phrase I still use: creative until proven otherwise. A choir member suggested an idea she was sure was original; turned out the tool already existed. Creativity, it seems, is just knowledge you happen not to have yet - which is uncomfortably true for humans and machines alike.
The verdict.
ChatGPT is a brilliant add-on and a terrible substitute. It can accelerate user research, but it cannot replace the moment a real person surprises you. You still have to talk to the humans. In its own words, quoted in the paper, the model needs the band to jam with - it is not meant to hog the spotlight solo.
A note on framing: this was a graded course paper, written solo and hosted on arXiv. Not peer-reviewed. It is playful on purpose - the title is a whole bit - but the method is real and the finding has held up.