Disclaimer: this case study was NOT written by AI. Honest.
Every great project begins with a great plan, and I'll be the first to admit I don't follow this advice as much as I should - I end up jumping into projects hot-headed, and pay the price later on. This is also the reality of the software business, where projects are conceived quickly, enthusiasm is high, and money is budgeted enthusiastically. So it was refreshing to be commissioned for a study that would be the cornerstone for something big down the line. The goal - learn more about the hardcore gamers of a particularly hardcore genre.
The study was commissioned both to serve as an element of a knowledge database that one of the largest gaming companies in the world maintained, and to support ongoing projects, either early in the development phase or already in beta. One game in particular was singled out as having a hardcore playerbase that would be great for in-depth interviews. The stimuli were the game itself, plus a short demo of another, similar title.
Getting real pro players is a job for a pro
Hardcore gamers are a tough breed to catch. When I recruited professionals for other studies - be it lawyers, doctors, or other specialists - it was fairly easy to confirm their bona fides via diplomas, certifications, and online profiles. This is not the case for gamers, who almost always go online by their nickname. The only way to confirm they're the real deal is to check their gamer score, a unique value they build over time on their game profile.
The reason this was problematic is something common in modern research recruitment: fraud. Earlier, it was decided the study would take place in Los Angeles. There are a number of great reasons for this - first of all, being a major city, Los Angeles has a huge population of gamers from all walks of life. There's also a business reason, as most global game companies have offices there, and that's because many of the resources needed to make games come from the movie industry - primarily actors and visual effects studios. California is also expensive, and everything comes at a premium, especially respondents for a study with a long LOI. If there's a way to earn money, people will go for it, and sometimes they'll stretch reality a bit to get there.
Here it's worth mentioning that consumer panels are not very good at getting actual gamers. Sure, they ask the right questions and many folks will give answers, but while you'll get a good range of casual gamers, your options for getting hardcore players are limited. I've encountered this in Germany before, and while there are specialized panels, they are expensive, and that tab is unsustainable for a cost-conscious customer. What I found is that gaming moderators have great personal networks, and reaching out to local university groups is an effective way of filling the gap. However, at the end of it all, you're still stuck needing to verify that they are who they say they are.
During the study we quickly noticed many players were not the pros they claimed to be. This was apparent in things as little as the way they placed their hand on the mouse and keyboard, or in very particular actions they took in-game. Pro players have extremely honed skills that show in their twitch reflexes and optimized hand and body stances. People who've played a particular section of a game will go for unobvious but optimized ways of reaching a goal faster. If you want to see this in action, I suggest watching a few speedrunning marathons for something like Skyrim.
Since we noticed this issue quickly, we had to adapt the study flow. During recruitment, players were required to send a screenshot of their gamer profile. We added a requirement to log in to their player profiles on the lab computer with the moderator close by, and the 2FA requirements usually meant they weren't able to, because that profile actually didn't belong to them. It still wasn't great, because the long LOI meant the time slot was essentially lost, and the incentive cost was eaten by us and the recruitment partner.
As you can see, I'm not tooting my own horn. It was a learning experience, and it took three of us to effectively spot all these little things and weed out the fake experts. I think being a pro is all about recognizing those mistakes, learning from them, and sharing with others. In this case, we could have probably saved a few hundred dollars and some time.
Help, I need a sweet rig NOW
All the game companies I've worked with are very particular about their hardware requirements. To be perfectly honest, this seems to be more of a rock star rider situation, where musicians would ask for something absurd in their dressing rooms before a concert - like Skittles sorted by colour - because if that was handled by the promoter, they could be assured the other, more important things had been handled as well.
Unless it's a console, your test machine has to be something that's at least on recommended specs, and ideally above them by a margin, though you won't need a 4090 graphics card. The overhead is there to guarantee a smooth gaming experience that won't bring false positives - if a game is jerky on an otherwise smooth-running machine, that could be a bug or a perception issue. Additionally, you'll usually run streaming or other video-recording equipment, and that brings a noticeable CPU and GPU load.
These unique requirements are more of an issue if you're fielding far from home. Based in central Europe, I've had to field studies in the Middle East, Brazil, and the US, and finding companies with that type of equipment is challenging, as hardware renters most often carry office equipment. There are some e-sports events companies that can help you, but as usual, that comes at a hefty premium.
There's also a legal issue - in this case, my employer had a US entity (and a US credit card!) that could secure a rental contract, and the deposit required wasn't as high. In other cases, you might be required to pay the whole rental fee up front, which, if you've run research, can be problematic, as most studies are paid for on results delivery.
Speaking of costs, there's an additional ballooning-cost problem. While in this case we only needed one, albeit strong, machine, sometimes a focus group test will require you to bring sixteen computers plus a local server machine, and each one can be around 200-300 USD per day (including gaming-grade monitors and accessories), with the rental contract signed on a per-week basis. Expensive! Ultimately, in this case, we found a great solution: our moderator was looking to buy a new monster rig for himself, so we came to a deal where he'd buy it and we'd rent it from him, essentially recouping most of his costs.
In a final equipment note, do take care of your internet connection. In this case, we fielded at the fantastic Adler-Weiner location on Wilshire Blvd., where the bandwidth was more than adequate for the double streaming - one stream for the game itself, and the other for beaming the game video to the customer's offices in various places around the world. With more computers, things become tricky, which I'll touch upon in another case study.
The game's not the most fun part
I'd love to share more about the game itself, and what we found, but honestly, in this case, it was quite unremarkable. Don't get me wrong - there were a lot of findings and hundreds of hours of unique gameplay to go through, but in this particular case the final report was handled in-house, and the most creative part I did was the fielding and wrangling the customer on the extremely strict screener. Just like a fresh medical doctor tends to find cancer in every new patient, researchers fall into the trap of thinking every study will bring a revolutionary finding, and I always say "no new findings is also a finding". They don't all have to be profound - and that goes for my sayings, too.