|
|
|
|
GOR 2001 Homepage
Indices
Management - Staff only
|
GOR 2001 - contentThis is the http://kiwi.uni-psych.gwdg.de/congress/gor-2001/contrib/buchanan-tom/buchanan-tom Document. Main Author: Buchanan, Tom Co-Authors: Reips, Ulf-Dietrich; Institution: Department of Psychology, University of Westminster, London Contribution Title: Platform-dependent biases in Online Research: Do Mac users really think different? Authors Email: tb99@iname.com URLs:
Abstract English (version: 25/06/2002 - 07:47, size: 835) English: The technologies and software used in online research are becoming increasingly sophisticated and innovative (e.g. Web experiments written in Authorware, Java applets). However, these developments mean that potential participants unable or unwilling to run such software will be excluded. If these people differ psychologically from those who are able to participate, samples obtained using these technologies will inevitably incorporate biases. To identify such potential biases, the scores of respondents using different hardware (PC, Mac, Other) and software (Javascript enabled or not) configurations when accessing an online personality inventory were compared. Findings indicated that there are personality differences between different groups of respondents. Implications of these results for online research will be discussed. Article (version: 25/06/2002 - 07:47, size: 22686) 1. Introduction In the first few years of online research, there have been rapid advances in a number of areas. Some of this research has been devoted to an exploration of the critical question of whether the Internet is an appropriate medium for psychological research. Such explorations of validity issues have on the whole been encouraging: Krantz & Dalal (2000), for instance, provide a review of studies which permit comparison of online and offline work, and suggest that by and large we can do valid research online.
Other types of studies have looked at online research methodology: what are the strengths and weaknesses of different techniques, what are the advantages and potential problems we may encounter? Dillman, Tortora, Conradt and Bowker (1998), for example, have shown that it is best to keep online questionnaires as simple as possible to reduce dropout rate. Musch, Broeder and Klauer (2001) have shown that the "randomised response technique" can be used to control for the possibility that people might give misleading answers to sensitive questions. Reips (2000) lists some of the strengths and weaknesses of online research, and suggests techniques for overcoming some of the problems.
Studies are also being published which move beyond methodological issues, and focus on important psychological phenomena. For these, the online environment may simply be the medium and not the focus of research (though there will indubitably be many for which it is both). An example is a study by Rogers et al (in press) looking at links between drug use and self-reported memory problems. This has supported laboratory findings with data from the much larger samples obtainable via the Internet.
As well as advances in psychological aspects of online research, there have been technological advances. As hardware and software develop, the range of techniques and tools we can use grows, and there is increasing sophistication in web research techniques.
While most web experiments used to be conducted using basic HTML and CGI combinations (and many still are) there are now many other techniques which have been adopted by online researchers. These often allow better visual presentation and more interactivity, achieving more control over the experimental environment through technologies such as Javascript, Java applets, and systems such as the Macromedia Authorware Web Player (formerly called Shockwave). All of these have been used in online research projects.
Many web researchers have already capitalised on the technological opportunities available to us. This is clearly a sign of progress, as the field develops. However, it may also bring problems.
The key question is whether or not all potential participants will have the software configurations required to participate in the study. In most cases, they probably will – but if an experiment requires the user’s browser to interpret anything other than plain HTML, there is a risk that some people will not be able to participate or will have a different experience of the experiment in terms of stimulus presentation (e.g. Burt et al, 2001).
Research using advanced techniques makes the assumption that everyone has the appropriate software available, or that if not then they are willing and able to install it (e.g. download plugins and players). There are ways of doing this relatively easily: for instance, studies employing Javascript based routines can screen people out or request that they use Javascript-enabled browsers. However, there are some (admittedly few) browsers which cannot handle Javascript, and there are some web users who prefer to disable Javascript in their browsers. This might be done for security reasons, to speed up their browsing, or to prevent annoying page redirections and popup windows. Might it be the case that people who use Javascript enabled browsers differ psychologically in some way (perhaps less security conscious, or less reluctant to hand control of their browser to a website) from those who do not?
As another example, some studies make use of video or audio players (e.g. the Authorware web player). These will advise potential participants that they need to install the relevant player. Again, this is easily done – the software can usually be downloaded freely.
However, some people may be unwilling to do this. Download times are frequently long (for example, at the time of writing the Authorware Web Player takes 11 minutes to download on a 56K modem – assuming optimal conditions). Depending on participants’ patience, the cost to them of being online for so long, level of confidence in their ability to install the player, and fear of introducing a virus or otherwise harming their system, some people may well choose not to download and install such players.
As well as those who are unwilling, there may be others who are unable to download and install players. This may be due to factors such as their connection speed, the power of the machine they are using, corporate or institutional policies (many participants in online studies are students sitting in computer labs – and educational institutions frequently forbid installation of software on classroom machines), and availability for their platform. At the time of writing, for instance, the Authorware Web Player was not available for all platforms. There was no EPOC version, and versions available for some other platforms (e.g. Linux and some Macs) were only compatible with Netscape and not other browsers – so one might have to install a different browser to participate in a study, not just the plugin!
Therefore, the possibility arises that by using advanced techniques one is preventing some potential participants from taking part in one’s study. Given the huge numbers of participants potentially available, though, does it matter if a few are screened out? It might.
Whether or not there is a problem will probably depend on the reasons for exclusion, especially if these are psychological in nature. If those who participate are psychologically different from those who might wish to but cannot, biases may be introduced which could affect experimental outcomes.
A related issue is the possibility that there might be psychological differences between different groups of computer users. If an experiment relies on software which will not run on a particular computer platform or operating system (as in the case of the web player described above), and if users of that system (who are thus excluded from participation) are in some way psychologically different from users of other systems, bias may again be introduced.
At least one computer company has made a lot of money from the suggestion that their users of their machines are different: Apple’s highly successful “Think Different” marketing campaign introduced the idea that Macintosh users were somehow different and more creative, by using advertising images of artists, athletes, scientists and various exceptional people. As well as such marketing ploys, there have been suggestions in the literature (e.g. Turkle, 1995) that Macintosh and PC users did differ in the way they thought and preferred to interact with their machines. Such suggestions are mainly anecdotal in nature, but the suggestion that PC and Macintosh users might differ in personality is worthy of exploration.
The goals of this project were, accordingly, to investigate (1) whether users of different computing platforms differed in personality, and (2) whether people using Javascript-enabled browsers differed from those unable or unwilling to use Javascript.
2. Method
The instrument used to assess personality was an existing online Five Factor personality inventory (Buchanan, Goldberg & Johnson, 1999) which provides indices of the five domains of the Five Factor Model (as described by Costa and McCrae, 1992) - Openness, Conscientiousness, Extraversion, Agreeableness and Neuroticism. The inventory has 42 items and is based on an IPIP inventory developed by Goldberg (1999). The five scales are acceptable in terms of internal consistency, and there is some preliminary evidence (based on correlations with self-reports of relevant behaviour) of validity.
For the purposes of the current study, a few additional questions were added to the end of the existing questionnaire. These included demographic information, and a question about the type of computer the respondent was using (PC, Mac, Other, Don’t Know). Participants were also asked whether their data could be used in analysis (so that people who did not consent, or who had not answered the questions seriously, could indicate that this was the case).
Participants found the inventory either via search engines or through links from other sites. Having completed the inventory, they receive instant feedback on their scores. Recruiting for this study was thus completely passive: no attempts were made to solicit volunteers; these were people who were actively looking for a personality test to complete. During the period of this study, an average of ten to fifteen people were completing the inventory every day.
The study was set up so that Javascript users and non-users were automatically assigned to different conditions. On arrival at the first page, a Javascript routine immediately redirected participants to an informed consent page. Those participants who did not have Javascript enabled obviously were not redirected. They instead saw a message saying the page had moved and asking them to click on a provided link. That link took them to another informed consent page. People entering through each of these two routes had an identical experience from that point on: the only difference between the two conditions was the datafile in which their responses were eventually logged. Having indicated their consent, participants were presented with the inventory and additional questions. On submitting their data, they were presented with an automatically generated description of their scores and a debriefing page.
3. Results
Prior to analysis, the dataset was screened for possible multiple submissions or data unusable for other reasons. 526 submissions were excluded because they came from IP addresses from which a response had previously been recorded, and might thus be duplicate submissions from the same person. 253 submissions were excluded because the responded had indicated that their data should not be used for analysis. Finally, 4 submissions were excluded because of implausible patterns of demographic responses (e.g. education levels which did not match their responses. Following this screening, 2148 apparently valid responses from unique IP addresses remained.
These comprised 865 men (40.3% of sample) and 1283 women (59.7% of sample). The most commonly reported location was the USA (51.8%), followed by Europe (31.0%). The largest proportion (46.9%) was employed for wages. There was also a considerable number who were students (41%). In terms of the main variables of interest, the majority of participants were PC users using Javascript enabled browsers. 1626 reported using PCs, 114 indicated that they were using a Mac, 211 were using another (unspecified) platform and 197 reported that they did not know what type of computer they were using. 1996 were using Javascript enabled browsers and 152 were not.
Javascript users and non-users did not differ significantly on any of the five personality dimensions assessed. They did however differ in level of education: a Mann-Whitney test indicated that Javascript users were significantly less educated (U=133321.5, p<.021). The modal level of education for those using Javascript enabled browsers was “Some college/university”. The modal level of education for those whose browsers were not Javascript enabled was “College/University graduate”.
When users of different computing platforms were compared, it was found that people who reported using Macs and PCs differed on one personality dimension: Openness (t(1738) = 4.60, p < .0005). Mac users were significantly higher in Openness (Mean = 28.04, SD = 4.93) than PC users (Mean = 25.84, SD = 4.94). There were no other significant differences between users of different computers.
4. Discussion
The results of this study appear to indicate people who used Javascript differed in education level from those who did not, and that Macintosh and PC users differed in levels of Openness. How are these differences to be explained?
A possible explanation for the Javascript results makes the assumption that people deliberately turned off Javascript functionality in their browsers (whether for security reasons, or to speed up browsing). In order to turn off this functionality, one needs to know what Javascript is, and why one might wish to disable it. Then, assuming one knows that it can be turned off, one needs to know how to actually do so at a mechanical level – an option that is well hidden in some modern browsers. It seems reasonable that possession of this knowledge will be a function of length of experience with using the Internet, which in turn is likely to correlate with education: the more time one has spent in education, the more likely one is to have used computers at an “expert” level for a longer period of time.
The difference between PC and Macintosh users is somewhat more interesting. People who score low on the dimension of Openness can be characterised as “down to earth”. They may be conventional and conservative in behaviour and outlook, less interested in art and culture, more practical in nature. High scorers, on the other hand, are likely to be more imaginative, more creative, more likely to seek out cultural and educational experiences. The fact that Macintosh users tended to score higher is consistent with both popular stereotypes and Apple’s marketing campaign – people low in Openness might be reasonably be expected to be less likely to “Think Different”. Might they thus be less likely to choose a Macintosh computer?
Another possibility is that some characteristics of the hardware and software being used influenced the way in which people responded. Despite some degree of convergence, the “look and feel” of PCs and Macs are still different (compare, for instance, the appearance of a colourful, futuristic looking iMac with a conventional desktop PC).
A third possibility is that the difference is an artefact. Macintosh computers are frequently used in creative, artistic settings: they are, for instance, well suited to art and design applications, manipulation of graphics, editing music and video. Therefore, people working in such settings—whom one might expect to be high in Openness—might be likely to end up using Macs rather than PCs, simply because those are the machines used in their workplace (or study environment). It could therefore be that more Open people simply happened to participate using Macs, not PCs, because those were the machines sitting on their desks.
It is clear that further work is required to uncover the reasons for the PC-Mac difference. However, whatever the ultimate explanation for the differences observed, the fact that there were differences at all has implications for online research.
It seems clear that if the current study had been accessible only to people with Javascript enabled browsers, or required software which would run only on a PC and not a Macintosh, a number of potential participants would have been excluded. Furthermore, there is evidence that the people who would have been excluded differed significantly from the rest of the sample in ways that could conceivably affect experimental outcomes. The possibility of technologically-caused biases seems to be a very real one.
In some circumstances, biases may not cause problems. For instance, Schwarz and Reips (1999) found no difference in results obtained with parallel CGI and Javascript versions of a study on the hindsight bias (despite a 13% higher dropout rate in the CGI condition). In other circumstances, they might be very problematic. Any work which seeks to obtain samples representative of some larger population will need to ensure that sections of that larger population are not exclude for technical reasons. Any work which relies on reference to norms (if that is in fact a meaningful concept in any non-finite Internet population) must take account of the fact that different norms might be needed for participants with different hardware and software setups.
The studies most likely to be at risk of problems are those where the phenomenon under investigation is linked to the construct with respect to which there is bias. For example, there are indications that Openness correlates with use of illicit drugs--people high in Openness are more likely to take substance-related risks (Furnham & Heaven, 1999). If a study like that of Rogers et al (in press) which seeks to recruit from drug-using populations excludes people high in Openness, a substantial proportion of the population of interest may be excluded. Similarly, Openness has been found to correlate with concern for social welfare (Furnham & Heaven, 1999). Some online studies have asked about issues such as donations to charity. Had they excluded people particularly high or low on Openness, could the results have been affected?
There are “real world” implications as well, especially with the growing use of online psychometric tests for personnel recruitment and selection. Imagine the scenario where an online recruitment agency wished to select applicants for a post in a creative industry where Openness was an important predictor of job performance—yet did this using a website which could not be properly viewed on a Macintosh. The most suitable candidates might thus be excluded from the selection process.
One might query whether there is actually a real problem here. With respect to the comparisons made in this study, almost all browsers nowadays are Javascript enabled, and almost all web content is accessible on both PC and Macintosh platforms. While the effect size (especially for the PC / Macintosh user comparison) is medium – of the order where it could importantly influence experimental results – relatively small numbers of people are affected by it. However, the fact that differences were found even for these comparisons suggests that in cases where potential participants are required to download and install players or other software, there may be bigger and more important differences. This is a question which can only be resolved, however, by further research.
It should also be noted that situations are likely to change over time, depending on the penetration of the technology used. At the time of writing, most web users will be using browsers capable of handling Javascript. Two years ago that might not have been the case. Obviously, the more people adopt a new technology the smaller the problem becomes. Other factors may also influence people’s willingness to download and install software – after a heavily publicised virus outbreak, for instance, they may be more cautious.
This study appears to have demonstrated that users of different hardware and software configurations can differ psychologically. It follows from this that if certain groups of users are excluded from participating in a study, the samples obtained in that study may be biased. Under some circumstances, this might affect core findings. We know of situations (parallel Javascript and CGI conditions) where findings have not been affected, but can envision situations where they might. On the basis of these observations, it seems prudent to recommend that online researchers should use the most basic and platform-independent technologies with which their research objectives can be achieved. There may well be times when the cutting edge of Internet technology is not the best tool for the job.
5. References
Buchanan, T., Goldberg, L.R., & Johnson, J.A. (1999, November). WWW personality assessment: Evaluation of an online Five Factor Inventory. Paper presented at the meeting of the Society for Computers in Psychology, Los Angeles, CA.
Burt, M., Little, A., Tiddeman, B., Penton-Volk, I., & Perrett, D. (2001). Experimenting with the web: Illustrative studies investigating facial attractiveness. Proceedings of The British Psychological Society, 9, 165.
Costa, P. T., Jr., & McCrae, R. R. (1992). Revised NEO Personality Inventory (NEO-PI-R) and NEO Five-Factor Inventory (NEO-FFI): Professional Manual. Odessa, FL: Psychological Assessment Resources.
Dillman, D. A., Tortora, R. D., Conradt., J., & Bowker, D. (1998, August). Influence of Plain vs. Fancy design on response rates for web surveys. Paper presented at Joint Statistical Meetings, Dallas, Texas.
Furnham, A., & Heaven, P. (1999). Personality and Social Behaviour. London: Arnold.
Goldberg, L. R. (1999). The Development of Five-Factor Domain Scales from the IPIP Item Pool. [On-line]. Available URL: http://ipip.ori.org/ipip/memo.htm
Krantz, J. H., & Dalal, R. (2000). Validity of Web-based psychological research. In M. H. Birnbaum (Ed.), Psychological Experiments on the Internet (35-60). San Diego: Academic Press.
Musch, J., Broeder, A., & Klauer, Ch. (2001). Improving survey research on the World-Wide Web using the Randomized Response Technique. In U.-D. Reips & M. Bosnjak (Eds.). Dimensions of Internet Science, (179-192). Lengerich: Pabst Science Publishers.
Reips, U.-D. (2000). The Web experiment method: Advantages, disadvantages, and solutions. In M. H. Birnbaum (Ed.), Psychological Experiments on the Internet, (69-117). San Diego: Academic Press.
Rodgers, J, Buchanan, T, Scholey, A.B., Heffernan, T.M., Ling, J., & Parrott. A. (In Press). Differential effects of ecstasy and cannabis on self-reports of memory ability: a web-based study. Human Psychopharmacology (special edition: The neuropsychopharmacology of MDMA).
Schwarz, S., & Reips, U.-D. (2001). CGI versus Javascript: A Web experiment on the Reversed Hindsight Bias. In U.-D. Reips & M. Bosnjak (Eds.). Dimensions of Internet Science, (75-90). Lengerich: Pabst Science Publishers.
Turkle, S. (1995). Life on the Screen: Identity in the age of the Internet. New York: Simon & Schuster. |