This is an article of seven short chapters aimed at helping you design and execute a successful playtest.
Hopefully. after reading this you will have a better understanding of what is involved in executing a playtest and you will have learned some new playtesting methods.
What is playtesting and why it is important
Before we really get into it, I just want to shortly address what I consider to be a playtest, and why I think it is crucial to the development of games.
My definition of a playtest is ‘a controlled research session where-in players from outside the development team play the game under observation, with a specific goal for the game designer.’ It’s not a very smooth definition, but it should get the message across.
I think it is crucial to game development because well, this and this.
Let’s get started.
Chapter 01 Defining your goals
The first and most important part of your playtest takes place before you gather players and start playing. It is in this phase that you define the things you are going to be researching.
Possibly the most important thing to keep in mind during this preparation phase is what the goals for your playtest are. A playtest where you want to know if there are collision errors in your game will be completely different from a session where you want to know about balancing issues, or if the controls feel tight.
It is not only important to have considered your goals in passing while you storm ahead to playtesting, it is the very core of what you are about to do and it will influence everything you do from this point onward.
Let’s take some examples of possible playtesting scenario’s and their accompanying goals.
For these examples I have taken the game Grim.
Grim is a small multiplayer shooter developed by MorePolygons, a group of students from my year. It features only two maps, one main weapon and three ‘ability-type’ weapons. Yet let’s see how differing goals can influence our playtest, shall we?
In this playtest, we are looking at mapflow. Which is basically looking at how easy it is to move through the environment. See how the player backtracks to earlier parts to try and get a feel for how easy it is to reach an objective from different routes. In a real research into mapflow, players would repeat the same segment over and over again, testing jump distances, looking at when players can use sprints, etc.
In this playtest, we are looking at general physics and collision problems. I have cut out the most boring parts, but it mainly involves repeating the same jump that might be a problem over and over again, or walking into a wall for five minutes, or crouching under something and pressing the jump button a couple hundred times.
Notice how even though all the factors are the same, a slight difference in goal warrants completely different behavior from our playtesters.
Keep this in mind when formulating your goals, and successively when you go out to find playtesters.
These are all goals which are aimed at finding something that is wrong with your game, but what if you want to achieve a certain goal, and are trying to work towards that? For instance, what if you want someone to be able to complete a level in under two minutes, or you want someone to be scared by an enemy you’ve placed. Each of these goals requires a wildly different playtest set-up.
Make sure you have a clear view of what you want to achieve, or what you want to know after you’ve completed your playtest. Because setting up a proper playtest takes a lot of time and effort, and you do not want to discover at the end of it that all your preciously gathered data is useless.
Chapter 02 Defining your research questions
Once you have defined your goals, you should now have a pretty clear picture of what you want to achieve with your playtest. Next, it is important to define a couple of measurable and quantifiable research questions based on these goals. Don’t take this task too lightly, forming a good research question is a very difficult task and can steer your entire research in the wrong direction.
For instance, when my goal is to find collision errors in my game, two research questions could be:
- How many collision errors are there in my game?
- Where are the collision errors in my game?
While the answers to both these questions will give you information on the collision errors in your game, only the second one will give you an answer which will help you fix the problem.
It is important that you really go in-depth here, so you know exactly what to look for during your playtest.
By measurable and quantifiable research questions I mean that your questions need to support answers which can’t just be answered with ‘yes-or-no’ answers. A good way of starting a research question is using who-what-why-where-how. Not only because these usually force an answer which can’t be answered with yes or no, but also because they help you think about what you want to research. So instead of ‘are there collision issues in my map?’ (yes, there always are) you would get ‘where are the collision issues in my map?’. Or instead of ‘is the balancing of my game fair? ‘ you would get ‘How many zerglings can defeat one marine?’.
Something else you’ll notice in the above example is that I am being pretty specific, formulating a good research question is already helping me shape the coming playtest.
The final point in formulating a good research question is not only to have a question with a quantifiable outcome, but to be very, very specific. Let’s see how many factors can influence the outcome of an engagement of 5 marines vs. 7 zerglings in StarCraft II. First of all there’s upgrades, there are the usual attack and armor for both zerg and terran, followed by the unique upgrades for the infantry units themselves, combat shields and stim versus adrenal glands and metabolic boost for the terran and zerg respectively have an enormous impact on this engagement. Which brings me to micro, is the marine just standing there, or is he using stim and stutter-step micro, and are the zerg being ‘conga-lined’ in or is it a proper pack which splits up and surrounds the marine? Finally, what are the terrain features, is the marine backed against a wall, or in a choke behind minerals, or at the top of a ramp?
In the above video, you can see that even though the units in the three engagements are the same (5 marines vs 7 zerglings, no upgrades). And the micro (unit movement) was relatively consistent, the fact that the terrain was different had an already substantial impact on the outcome of the engagement.
You see how a relatively simple engagement like this can have a lot of factors determining it’s outcome. So lets try and formulate a research question that would allow us to execute this playtest for determining how many zerglings a pack of marines can take.
How many unupgraded zerglings can an unupgraded marine kill in flat terrain if the marine holds position and the zerglings have a 0.5 second interval between arrival?
You see how a relatively easy question can become a pretty complicated affair once you take a ‘scientific’ approach. But we really need this kind of accuracy if we’re going to execute a playtest which has any use.
Chapter 03 Determining your methods and tools
The next step to undertake is to determine which research methods and tools you will be employing during your playtests.
I have found there are two main methods of approaching playtests.
These methods are not mutually exclusive, you can mix and match different parts of the methods as much as you like, or rather, as much as is needed.
Method 01: Open
Using this method, you are fully open to the subjects, before you start playing you tell them the purpose of the playtest, what the game is, how the controls work etc. Also you let them know if you’ll be recording their session, you show the camera’s etc and if you work with one or more observant, they can walk around in full view, and take notes. Also you’ll answer any questions that arise from players, and you’ll ask the players questions during play.
Method 02: Closed
In this second method you provide the subjects with as little information as possible. You preferably don’t talk to them at all. If you can manage it, try to fit your game rules and controls on one piece of paper (if you can’t, there’s something wrong with your game design). Just try to show your subjects what is going on as little as possible, if you’re recording the playtest, don’t tell them beforehand and If you’re using observants, try to have them kept hidden.
As far as tools go, you want to be able to record as much of what is going on as possible, without generating so much data that you lose oversight. What you will be recording also depends wholly on what you are playtesting. For instance, if you are testing to see if there are any bugs in your game, you will want to record the player’s screen and some in-game metrics. But if you are interested in how players feel during certain segments of your game, you will want to record the player’s face and you will probably want to observe him personally.
These are some tools I personally use, and situations I use them for:
- Video camera + tripod
For capturing player activities where players are involved in physical manner, like prototypes for installations or paper prototypes.
- (HD) Webcam
For unobtrusively capturing player’s expressions while they play digital prototypes.
- Camtasia
For recording screen and webcam simultaneously. This really helps identify which in game actions correspond with certain emotions.
- Sound recorder
To record Q&A sessions so I can easily transcribe them later.
- Pen and paper
Always, always, always have pen and paper with you. There is no excuse.
- XML output
If at all possible, get your programmer to write something which will record in-game metrics such as characters’ position, health, times etc. and output it into some convenient file format such as xml. If the engine supports it you can use replay files.
- Questionnaires
During early playtesting I usually use general questionnaires to draw some general feedback out of players which will give me a direction to take with further playtesting. I also use questionnaires to help determine if I am succeeding at getting certain reactions from players.
Chapter 04 Preparing your playtest
The most important part during direct preparation of your playtest is to have a stable build of your game, and to have a clearly defined list of what features are implemented. It helps to make a document for each version of your game which states the game’s version number and then a list of implemented features and known bugs.
This helps you not only separate the new bugs during the playtest but knowing the state of the game during the playtest helps you make gameplay decisions if you come back to your data later.
Preparation for your playtest really depends on a lot of factors I described earlier, so there isn’t much I can say about it in a general sense apart from that you have to be really precise in your playtest. Make sure that if you set up camera’s or recording programs that they are in the right place, and that you can switch it all on with a single button press. Make sure you have pen and pad ready (again, there’s never any excuse to not have pen and pad ready). And maybe have some food and drinks for your playtesters if it is going to be a long session. The key here is to have the playtesters feel comfortable and to have them forget that it is a playtest as much as possible.
For me personally, it helps to make a little schematic of how the set up will be. This is especially usefull if you're working with others on the playtest.
If you're going to have a lot of playtesters, make sure you spread them out across a lot of time, taking your time to sit down with your playtesters can benefit the amount of feedback you gain from them. If they feel they should be leaving because the next playtester has arrived, they will probably not give you as much feedback as they would have otherwise.
Finding people to do your playtest should be pretty straightforward as long as you keep your target audience in mind. Make sure the players also fit your playtesting goals. For instance if you need to check whether your tutorial is clear enough, you’re going to have to find different players for each playtest. But if you’re trying to find collision issues or bugs in your games, you’re probably going to have to find players who are very experienced at playing your game, preferably even players who are very adept at breaking games.
As a final note, make sure you have your questionnaires printed and your instructions ready.
Chapter 05 Executing your playtest
If you’ve done your preparation right, actually executing your playtest should be a breeze. Just welcome the players, switch on the gear and sit back to observe and write everything down. Keep your methods in mind and you’ll be fine.
Playtest kolom from Ricky Raaskal on Vimeo.
This is an example of a playtest I executed a while back for an interactive installation I'm building with Raaskal.
The playtest was very succesful, and the video helped immensely with not only convincing the viewers that KOLOM was succesful at what it set out to do, but it also allowed us to review certain behaviours which were cropping up amongst players.
Chapter 06. Compiling results
Awesome, you’ve successfully executed a playtest! And now you have a hard drive full of screen and webcam footage, xml files from here to Tokyo and three notepad full of random scribbles like ‘COOKIES INCREASE IMMERSION’.
Now what.
Well, time to compile that mountain into something more manageable. Again, this phase is really dependant on what type of playtest you’ve done, but your main goal is to find trends in the data which will allow you to improve your game.
If your playtest was focused on bugs and gaps in your game, you might think it is sufficient to list all the bugs and gaps and have your programmer fix all items. But what if you consistently see a certain collision issue crop up? It might be more prudent at that stage to have your programmer look at the entire collision system in your engine.
Likewise, if players are consistently unimpressed by the big boss battle somewhere, it’s a sure sign you have to sit down with your artist to address this issue.
It is equally important to not only focus on the negative, if you see for instance that players always react heavily to the behavior of a certain enemy, be sure to note this down as well, and try and find ways you can exploit this strong reaction. Even better, try to distill why players react so heavily to this enemy and try and introduce these elements in other parts of your game.
I don’t always gather up data in graphs and statistics, I find the approach of going through all the data and focusing on the trends that jump out to me usually works more than well enough. If I have to communicate the results to other people on the team, I will on occasion summarize the trends in small infographics.
Chapter 07. Drawing conclusions
So, now you have a bunch of trends and things that stand out to you. Great! I haven’t got much to say on how you should deal with these trends because obviously they are way too specific per case for me to even scratch the surface of advice. All I can say on this subject is that you need to carefully record the steps you’ve taken to remedy any problems you’ve found through your playtest, so you can specifically playtest this fix, and see if it has worked.
Good luck and have fun with your games!