Why PEQs do not provide the best process evidence
The questionnaire (or self-administered survey) was invented by Sir Francis Galton, A British anthropologist, explorer, and statistician in the late 1800s. He used them as a cheap and standardized way to elicit people’s opinions. After an experiment, researchers often also distribute a questionnaire. Traditionally, the primary role of this post-experimental questionnaire (PEQ) has been an ancillary one. They have been used to elicit a handful of stable variables that help safeguard the quality of the data and verify randomization across treatments. The variables elicited by PEQs include manipulation checks, which are statements that help evaluate the degree to which the experimental stimuli worked, and demographic variables, which help test the extent to which treatments have been randomly allocated to participants.
However, in the experimental accounting literature, PEQs serve a much more critical role because many also use them to obtain process evidence. Process evidence is empirical evidence supporting, as narrowly and tightly as possible, the theoretical mechanism central to a hypothesis. Accounting researchers often require such evidence before accepting that the results generated by an experiment support a hypothesis. Although there are many ways that controlled experiments can obtain process evidence (see, for instance, Asay, Guggenmos, Kadous, Koonce, and Libby (2021) for an insightful discussion), PEQs, by far, seem the most popular way for experimental accounting researchers to try and obtain it.
There are two broad ways accounting researchers use PEQs to obtain process evidence for controlled experiments. First, PEQs may ask participants to reflect on their behavior and decisions during the experiment they just took part in. If participants’ reflections match the maintained theoretical explanation and explain part of the observed relationship predicted by the hypothesis, experimental researchers have produced process evidence. Second, PEQs may also ask participants to reflect on their personality and general attitudes in daily life. If their personality or general attitudes change (part of) the observed relationship predicted by the hypothesis in a conceptually consistent way, experimental researchers have also produced process evidence.
Although PEQs are a popular method for obtaining process evidence in the experimental accounting literature, using them has some significant drawbacks that are rarely discussed in the context of controlled experiments. Also, once you carefully consider the experimental method’s strengths, it should become clear that PEQs, at best, offer process evidence that is ancillary to the process evidence that controlled experiments can directly produce.
What are the problems?
PEQs invite socially desirable reporting
Social psychologists, who make great use of questionnaires, have long known about and fought their drawbacks. One of the most pronounced and discussed problems with questionnaires is respondents’ tendency to report socially desirable answers (Nederhof 1985; Podsakoff, MacKenzie, Lee, and Podsakoff 2003; De Jong, Pieters, and Fox 2010; Steenkamp, De Jong, and Baumgartner 2010). In the context of controlled experiments, socially desirable reporting means that once participants arrive at the PEQ, they will say what they think is socially desirable, which is a critical problem if the theory involves a socially sensitive issue (i.e., think about misreporting, honesty, and effort distortion).
Participants may experience fatigue
Accounting researchers often include an array of statements that jointly capture some stable personality trait or mindset among participants. Accordingly, PEQs tend to prolong the duration of controlled experiments substantially. However, participants’ attention span is limited and wears down as the experiment progresses, which is why many do not prolong the experiment more than they have to. When we use PEQs, the chances are relatively high that participants experience fatigue by the time they reach it. Accordingly, they may not fill out the PEQ seriously, leading to more uniform answers (Krosnick, 1999; Galesic and Bosnjak, 2009).
PEQs may implement false memories
Even if participants are still fresh and pay attention by the time they reach the PEQ, they may not perfectly recall what happened or how they thought and felt during the experiment. Ross (1989) and Loftus (1999) provide numerous examples of people reconstructing beliefs under such circumstances using implicit theories rather than relying on their imperfect memory of what actually occurred and what they actually believed at the time. There is also a trail of evidence that false memories and beliefs can arise, sometimes unintentionally, in people’s minds (Gilbert, Krull, and Malone 1990).
Participants may rationalize what happened
PEQs are, by definition, distributed after the experiment has been conducted. Therefore, everything that came before the PEQ influences how participants answer questions and evaluate statements in the PEQ. Suppose participants still have a detailed account in their mind of what happened and are sufficiently motivated to finish the PEQ. In that case, they may fall prey to rationalizing their recent behavior and decisions. In other words, when answering statements in PEQs, participants may resolve any cognitive dissonance by changing their minds about why they behaved the way they did . Suppose we ask participants whether some theoretical explanation and intervention drove their behavior and decisions during the experiment. In that case, it is likely they may agree, even if another explanation is more representative of the truth.
PEQs fail to detect changes in thinking
People often believe that what they are thinking at this moment in time represents how they have thought all along. Therefore, they tend to be unaware of changes in their thinking (Bem and McConnell, 1970). If a participant has changed their opinion during the experiment, they may not report this change in the PEQ. Therefore, using the PEQ to detect changes in thinking may lead to wrong conclusions. For instance, PEQs may reveal no evidence that changes in their thinking drive the changes in participant behavior caused by an intervention while actually, they did.
What about stable personality traits?
Proponents of using PEQs to obtain process evidence may point out that verified instruments measuring stable personality traits are less prone to the abovementioned issues. Precisely, these instruments try to measure participants’ personality type in daily life, which should be independent of what happened during the experiment. If the instrument has been verified and is sufficiently concise and internally consistent, it could provide an excellent way to verify the process underlying an observed relationship.
However, social psychologists have become increasingly concerned about using these instruments specifically and the concept of stable personality traits more generally. As measured by these instruments, many personality traits turn out not to be very stable, in particular, in terms of stability within people over time (Bardi and Zentner 2017; Golsteyn and Schildberg-Hörisch 2017). Some social psychologists have devoted entire careers to proving that the situation is a much more stable driver of people’s behavior than some individual typology (Ross and Nisbett 2011). Thus, even if we use verified personality instruments to showcase process evidence, the relative instability of participants’ responses make it challenging to draw reliable conclusions.
The experiment itself can provide the process evidence
Although questionnaires, outside the context of controlled experiments, give us a cheap way to gain insight into people’s thinking, they rarely provide robust causal evidence of theoretical relationships (Bloomfield, Nelson, and Soltes 2016). It turns out this particular weakness of questionnaires is precisely the comparative advantage of controlled experiments! Controlled experiments introduce interventions and manipulate independent variables with meticulous precision to see how they change participants’ behavior. Depending on how tightly we control the experimental setting, you can find relatively convincing evidence of the process underlying an observed relationship.
Controlled experiments can vary in their ability to provide process evidence because some grant more insight into the process underlying observed relationships than others. One convincing way that controlled experiments can directly obtain process evidence is by featuring interaction effects between different manipulated independent variables. Suppose one manipulation attenuates or strengthens the influence of another manipulation in a conceptually consistent way. In that case, we are surer our theoretical explanation drives the variation observed in the dependent variable(s). Therefore, this type of design we often coin a moderation-of-process design.
Another way that controlled experiments can directly obtain convincing process evidence is by unobtrusively observing participant behavior (Webb, Campbell, Schwartz, and Sechrest 1999). For instance, consider an experiment in which we measure how much time participants spend on a page. The time spent on a particular page could indicate how much attention participants give to a particular piece of information or stimuli. Depending on your hypothesis, it can be used to verify its underlying mechanism. For learning more about this approach to obtaining process evidence, please see this excellent post by Christian Peters.
