Automize testing your experiments with 'bots'
If you are familiar with programming applications for experiments, you have already learned about one of its most time-consuming and stressful phases: Testing the application. Testing applications of experiments is typically the last stage of development and is especially important for experimental research. One error or mistake in the code could mean the difference between a experiment that generates data you can use and one that generates unusable data.
The testing phase can be incredibly time-consuming and stressful because, in most cases, it requires you to click through the entire application multiple times, testing every option, choice, and possible outcome participants may face. The testing process also increases in length when your experiment includes multiple rounds and interaction between participants. However, what makes the testing phase particularly time-consuming and stressful is that if you decide to make one change afterward, you technically have to test the entire application again. Therefore, experimental researchers face a choice between rigorously testing their application for hours, if not days, or skip the testing process and run the risk of generating unusable data.
oTree, an open-source software package for experimental and survey research, offers a handy, but often overlooked, tool to test applications for experiments: Bots. If you are unfamiliar with oTree, learn about why you should be using oTree in this video. In essence, bots simulate participants and click through the application for you. You can program them to fill out forms and check whether everything works properly. If you are programming your oTree application yourself, you can program these bots using the “test.py” file in your application’s folder. In “test.py,” you can program any bot behavior you would like. oTree studio also offers a service that automatically generates the bot code for you, which is useful if you feel less confident with programming or if you do not have much time.
Bots’ capabilities do not restrict themselves to testing your application. Instead, you can also program different types of bots or program them to respond to a manipulation or intervention in a certain way. For instance, if one of your independent variables is a participant variable, you can create a different bot for each participant type. When the different bots click through your application, they will submit forms differently and make choices that depend on their type. You can also program bots to respond differently depending on the treatment they are in or on which page or information they have encountered. Generating and exporting the data generated by different bots enables you to analyze whether it is possible to test your hypothesis with your current application. If your theory is exceptionally detailed, you may even have priors about how bots should behave differently depending on their type or how strongly they should respond differently depending on your manipulations. By varying the number of bots, you can also assess how many participants you will need to detect the hypothesized effect if it exists.
Summary
Bots are a powerful yet often overlooked oTree feature that helps experimental researchers test their applications more effectively and efficiently. Yet, they also allow us to run relatively simple simulations. To be clear, the bot feature in oTree has not been created to run complex simulations. However, it may be meaningful for experimental research to complement the analysis of participant data with the analysis of simulated data both before and after the experiment is conducted. Please share your thoughts below. If you want more information about how to program these bots, please see the oTree documentation here.
