|
|
|
|
GOR 2001 Homepage
Indices
Management - Staff only
|
GOR 2001 - contentThis is the http://kiwi.uni-psych.gwdg.de/congress/gor-2001/contrib/contrib/silberer-guenter/contrib/eichstaedt-jan/eichstaedt-jan Document. Main Author: Eichstaedt, Jan Co-Authors: ; Institution: Universität der Bundeswehr, Hamburg Contribution Title: Technique and application of measuring reaction time using Java-Applets: How to improve online marketing research by social-cognition-based measures in Web experiments. Authors Email: jan.eichstaedt@unibw-hamburg.de Abstract German (version: 25/06/2002 - 07:47, size: 0) English: Without precaution Java-Applets are often too imprecise to reveal differences in reaction time. This can be settled by a new technique and gives way for measuring, e.g., implicit product attitudes by the implicit association test (IAT; Greenwald, McGhee, & Schwartz, 1998). Those and similar measures may give evidence of higher prognostic validity than classical explicit attitude measures in the field of certain product classes and market segments. Besides the technical aspects, the present paper will present 3 evaluation studies on the achieved precision of measuring reaction time via applets. Furthermore, the paper will present first applications of the new technique in the realm of marketing research. Article (version: 25/06/2002 - 07:47, size: 20582) Wouldn’t it be great? Early on Friday morning, you, a researcher, interested in reaction times, have a bright idea for a very telling experiment on latencies. As the idea might really be insightful, you want to know the outcome immediately. So you modify a JAVAapplet of the experiment's prior version to incorporate your new idea. It only takes you another half an hour to write down the short instruction in HTML and add an applet tag. Then you switch back to very old and puzzling data on verbal reports. After a nice day with qualitative data, your quantitative collection of the latencies is complete. The deviations look good. The differences in average latencies fit your hypotheses and the effect is huge and, of course, highly significant. Have a nice weekend! Unfortunately so far, this is only a fairy tale, because of the inaccuracy of latencies measured with applets. It is much more probable that the latencies actually measured by the applet are so inaccurate that everything looks like a mess. There are quite a few studies using Internet technique on measures other than latencies (e.g. Janetzko, Hildebrandt, & Meyer, 1999; Klauer, Musch, & Naumer, 2000; Krantz, Ballard, & Scher, 1997; Reips, 1996; 2000; Welch & Krantz, 1996). To measure latencies, JAVAapplets are a step in the right direction, because they run on client side and therefore no net lag can obscure latencies. At first glance this enables us to measure social-cognition-based measures like implicit product attitudes by the implicit association test (IAT; Greenwald, McGhee, & Schwartz, 1998), which is based on latency measurement. The IAT would be of great virtue for online marketing research. Participants of an IAT categorize stimuli of two different categories or domains, each in two subcategories, using only two response keys. Because one of the categories consists of positive or negative adjectives and the other consists of the attitude objects, subcategory memberships can correspond to an evaluative dimension. If they are compatible, categorizing will be fast. However, if they are incompatible, categorizing will be slow and the latency difference between compatible and incompatible condition will yield a compatibility effect. This compatibility effect is used to indirectly measure the participant's attitude. Some researchers hope that implicit attitudes measured by the IAT will have higher prognostic validity than classical explicit attitude measures in the field of marketing research (Plessner, Richter, & Wänke, 2000; Brunel, Collins, Greenwald, and Tietje, 1999; Maison, Greenwald, & Bruin, in press). In the domains of prejudice (Greenwald, McGhee, & Schwartz, 1998), self-esteem (Greewald & Farnham, 2000) or gender stereotypes (Nosek, Banaij, & Greenwald, 2000) the IAT has yielded promising results. Kim and Greenwald (1998) as well as Banse, Seise, and Zerbes (2001) provide evidence that participants of the IAT are not able to intentionally fake the measure. Thus, measuring consumer attitudes by the IAT would reveal the real attitude, even if the consumer couldn't verbally report on it or refused to do so. However, in most of the cited studies, strong valences of the attitude objects have been involved. Consumer attitudes may be much weaker and therefore the compatibility effect may be much smaller, too. So the inaccuracy of latencies measured by JAVAapplets may hinder the realization of this compelling chance for marketing research in the Internet. Chiefly, the inaccurate-timing problem lies in the ability of modern operating systems to do multi tasking. It might be possible that an implementation, which is accurate under normal conditions, becomes inaccurate when, for example, the mail client (running in addition to the web-browser and the virtual machine) begins to download an attachment. Or the browser, while displaying the experiment-program, may simultaneously request another page. The processor may become overloaded, if that second page contains elements which take up a lot of resources, like an animated GIF, another JAVAapplet etc. Since such activity cannot be controlled, applets which implement experiments on the Internet must be able to prevent the impairment of the measures which is provoked by such multi tasking. One way is to check whether the client system's performance is high enough. After successfully passing this test, latencies are recorded without further checking. Hecht, Oesker, Kaiser, Civelek, and Stecker (1999) have used this approach, although new sources of interference may arise later in the procedure. An ongoing check is desirable, but such checks consume valuable CPU-time and therefore may interfere with time-critical processes. However, Janetzko, Hildebrandt, and Meyer (1999; Hildebrandt, Meyer, & Janetzko, 1998) tested each measurement, even at later stages of the experiment. The inaccurate-timing filter identifies impaired measurements by testing whether the virtual machine makes false measurements of a sleep interval of, let's say, 50 ms. In order to initiate the sleep interval, the sleep(d) method is issued on the control thread, which suspends it for the specified duration d. Consequently, it takes up virtually no processing power. Nevertheless, the virtual machine measures, by a specialized method, whether the duration d is exceeded. Additionally, the filter records start time and end time of the sleep interval by another method. If everything functions well, end time minus start time has to be 50 ms. If something interferes, one of the two software clocks ought to be more affected than the other. This in turn leads to inconsistencies in the measurement of the sleep interval, which the filter detects. The control thread runs in parallel to the normal business of recording the latencies and testing whether the latency measurement is correct. Compared to Janetzko's et al. parallel evaluation of the system's capacity, the filter tests the system's ability to measure time intervals directly, rather than drawing on the system's global performance. So only the measurement of the sleep interval is iterated again and again until the latency is recorded. As the control thread measures nothing other than sleep intervals, it is designed to take up very little processing power, in order to avoid interference with the latencies' measurement. Measurements of the sleep interval that the control thread identifies as false are ruled out, thus preventing corruption of the reaction time’s statistics. However, the filter is not perfect and can never correct a false measurement. The implemented applet runs on all computer systems with full support of JDK 1.1 (Sun Microsystems, 1997). No additional permission by the client is required. To summarize, the approach is to give the virtual machine a chance to yield false time measurements by running two different software clocks at the same time. The inaccurate-timing filter was evaluated in three studies (Eichstaedt, in press) using 11 different client systems. The keystroke repetition rate was used as an external input of latencies at a reliable and known rate. The first study showed a much higher precision of native programs on computers in the laboratory compared to JAVAapplet-based measurement, even without any impairment by multi tasking. However, when using the inaccurate-timing filter, standard deviation of the latencies became substantially reduced compared to JAVAapplets without the filter. Nevertheless, native programs remained much more reliable than JAVAapplets, even when equipped with the filter. Thus, when possible, the laboratory is the better choice. But what, if there is no way round the Internet? The second study investigated what could happen in the worst case. The applet was running while calling additional web pages or running further applications in parallel. Errors in measuring reaction time were very much increased when the computer system ran additional processes. However, the filter diminished the standard deviation of the reaction time drastically by ruling out false measurements in all but one computer system. In order to test whether the filter would help to find differences in arithmetic means of latencies which would otherwise be missed, the third study compared different keystroke repetition rates. Without filter, the different keystroke repetition rates were more difficult to identify than with filter. The decline in standard deviation showed up again, when using the filter compared to not using it. On the other hand, this had a price, as the number of accepted measurements diminished greatly. Thus, the inaccurate-timing filter might enable applets to be used as the basis for latency measurement experiments with an accuracy previously not met by applets. On the other hand, differences in keystroke repetitions do not provide a good example with which real effects in the cognitive domain can be compared. This is due to the minimal variance of the keystroke differences' underlying process, which is of electronic rather than psychological origin. Therefore using a more relevant effect is desirable. The best candidate may be the compatibility effect obtained by the IAT, because it would demonstrate the inaccurate-timing filter's performance in a straight experiment which, nevertheless, allows us to use the filter in the field of consumer attitudes. Experiment 1 Greenwald's et al. (1998) first experiment was repeated. So the question remains the same. Would the IAT validely reveal attitudes for objects of intersubjectively obvious valences? Compared to Greenwald's et al procedure, the extension is to provide two classes of attitude objects marking two grades of valences. So the valence extremity was varied as one factor in the design to yield attitudes of two extremities, which were supposed to be reflected in the IAT-measure. If this is the result, it provides evidence that the IAT may be capable of reflecting attitude extremity (Abelson, 1995; Downing, Judd, & Brauer, 1992; Tesser & Mendolia, 1995). More important for the technical purpose of this paper, the lower attitude extremity of the two should show its compatibility effect only with the help of the inaccurate-timing filter. Therefore corresponding planned comparisons were calculated. Procedure. A Web-experiment was conducted by means of a JAVAapplet which applied the inaccurate-timing filter. The client computers were brought in the study by the participants. Also the potential interference of additional programs running in multi tasking was not controllable. In two successive IATs the participants had to distinguish pictures of spiders from pictures of butterflies in the category insects and smiling faces from angry faces in the category faces. A fixed RSI of 1000 ms was used. In the first step of the IAT-procedure each participant had to categorize the attitude objects into the two subcategories. This was done in two blocks, each containing 50 trials (category decisions). In the second step participants had to decide whether an adjective was positive or negative. In the third block both tasks, categorization and adjective-decision, followed in alternating succession in four blocks. The fourth step reversed the association of the subcategories to the answering keys in one block. The fifth step mirrored the third and used the key association of the fourth step for the categorization of the attitude objects. Then a second IAT-procedure followed, using the second category which omitted the obsolete second step (see Greenwald et al., 1998, Exp. 1, for details). 20 students of the University of the Federal Armed Forces Hamburg participated and got credit points for it. Results. For sake of brevity, only the compatibility and the category effects are reported. Both main effects are highly significant with means in the predicted directions (fig. 1), supporting the hypotheses.
However, only the planned comparison within the more extreme faces-category is significant. The less extreme insect category did not yield a significant compatibility effect (F(1,19)=4.279, p < .10). This changed when the inaccurate-timing filter was applied which allowed for significance (see fig. 1) also with insects as attitude objects. The difference in this category is only 40 ms., but the inaccurate-timing filter reveals this difference to be small but reliable. Experiment 2 Are product attitudes accessible with the same validity as those obvious object attitudes of Experiment 1? One product was chosen. In the standard version the IAT can only measure the attitude difference of two subcategories in comparison. This is a disadvantage, since explicit measures all are capable of measuring absolute attitudes of single objects. Procedure. In order to measure the product attitude of only one product pictures of the product were taken before and after use of this product. Specifically, a box of After Eight chocolates was photographed in an enticing manner with all those fine chocolate mints included and displayed (new After Eight) and, as a counter part, the leftovers of the packaging with not quite so enticing chocolate mint fragments were pictured (used After Eight). Obviously, attitudes should be more positive for new After Eight than for used After Eight. The material was run in an IAT with 10 students participating. Results. There was a main effect compatibility (F(1,9)=12.376; p< .01) with 655 ms in the compatible condition and 755 ms in the incompatible condition, substantiating that Greenwald's et al. results are replicable with a product attitude, too, thereby revealing an attitude for a single attitude object. Without the inaccurate-timing filter the compatibility effect missed the standard significance criterion (F(1,9)= 4.405575; p= .065). Discussion As regards technique, the inaccurate-timing filter did its job well enough to find existing effects which otherwise would have been missed. This enables us to apply the IAT in the field of online marketing research. As regards content, the results of Experiment 1 provide evidence that the IAT may be capable of depicting attitude extremity. Experiment 2's results show that product attitudes for a single product also are accessible. It would be interesting to compare the new/used-approach taken here with the go-nogo method of (Nosek & Banaji, submitted), which also allows us to measure attitudes for single objects. Measuring absolute attitudes seems to be inevitable in the realm of marketing research, because comparison of more than two products is necessary. So we have a context-free measurement of product attitudes. While the inaccurate-timing filter was applied to the IAT-procedure, it can also be useful in all application where latency effects are not expected to be huge but where, at the same time, the internet must be used. References Abelson, R.P. (1995). Attitude extremity. In: R.E. Petty & J.A. Krosnick (Eds). Attitude strength: Antecedents and consequences. Ohio State University series on attitudes and persuasion, Vol. 4. (pp. 25-41). Hillsdale, NJ: Lawrence Erlbaum Associates. Banse, R., Seise, J., & Zerbes, N. (2001). Implicit attitudes towards homosexuality: Reliability, validity, and controllability of the IAT. Zeitschrift für Experimentelle Psychologie, 48, 145-160. Brunel, F.F., Collins, C.M., Greenwald, A.G. & Tietje, B.C. (1999, October). Making the private public, accessing the inaccessible: Marketing applications of the Implicit Association Test. Paper presented at meetings of the Association for Consumer Research, Columbus, Ohio. Downing, J.W., Judd, C.M., & Brauer, M. (1992). Effects of repeated expressions on attitude extremity. Journal of Personality and Social Psychology, 63, 17-29. Eichstaedt, J. (in press). An inaccurate-timing filter for reaction-time measurement by JAVA-applets implementing Internet-based experiments. Behavior Research Methods, Instruments, & Computers, 33. Greenwald, A.G., McGhee, D.E. & Schwartz, J.L.K. (1998). Measuring individual differences in implicit Cognition: The implicit association test. Journal of Personality and Social Psychology, 74, 1464-1480. Greenwald, A.G. & Farmham, S.D. (2000). Using the Implicit Association Test to measure self-esteem and self-concept. Journal of Personality and Social Psychology, 79, 1002-1038. Hecht, H., Oesker, M., Kaiser, A., Civilek, H., & Stecker, T. (1999). A perception experiment with time-critical graphics animation on the word-wide web. Behavior Research Methods, Instruments, & Computers, 31, 439-445. Hildebrandt, M., Meyer, H. & Janetzko, D. (1998, October). JAVA-basierte Versuchsdurchführung: Potentiale und Probleme am Beispiel von Reaktionszeitexperimenten. [JAVA-based conducting of experiments: Potentials and problems shown by the example of reaction time experiments]. Paper read at German Online Research (28.-29. 10.), Mannheim. Janetzko, D., Hildebrandt, M., & Meyer, H. (1999, March). JAVA-basierte Versuchsdurchführung im WWW: Potentiale und Probleme am Beispiel von Reaktionszeiten. [JAVA-based experimenting in the WWW: Potentials and problems shown by the example of reaction times]. Abstract in: E. Schröger, A. Mecklinger, & A. Widmann (Eds.). Experimentelle Psychologie: 41. Tagung experimentell arbeitender Psychologen. Lengerich: Pabst. Kim, D.-Y. & Greenwald, A.G. (1998, May). Voluntary controllability of implicit cognition: can implicit attitudes be faked? Paper presented at meetings of the Midwestern Psychological Association, Chicago. Klauer, K.C., Musch, J. & Naumer, B. (2000). On belief bias in syllogistic reasoning. Psychological Review, 107, 852-884. Krantz, J.H., Ballard, J., & Scher, J. (1997). Comparing the results of laboratory and Word-Wide Web samples on the determinants of female attractiveness. Behavior Research Methods, Instruments, & Computers, 29, 264-269. Maison, D., Geenwald, A.G., & Bruin, R. (in press). The Implicit Association Test as a measure of implicit consumer attitudes. Polish Psychological Bulletin. Plessner, H., Richter, L. & Wänke, M. (2000, September). Implizite Einstellungen als Prädiktoren von Konsumverhalten: Anwendungen des Implicit Association Test. Vortrag auf dem 41. Kongreß der Deutschen Gesellschaft für Psychologie in Jena. Nosek, B. & Banaji, M.R. (submitted). The go/no-go association task. Manuscript submitted for publication. Yale University, New Haven, CT. Nosek, B., Banaji, M.R., & Greenwald, A.G. (2000). Math = Male, Me = Female, therefore Math ¹ Me. Manuscript submitted for publication. Yale University, New Haven, CT. Reips, U.-D. (1996, October). Experimenting in the World-Wide Web. Paper presented at the 1996 Society for Computers in Psychology conference, Chicago. Reips, U.-D. (2000). The Web Experiment Method: Advantages, Disadvantages, and Solutions. In M. H. Birnbaum (Ed.), Psychological Experiments on the Internet (pp. 89-114). San Diego, CA: Academic Press. Sun Microsystems (1997). JAVA development kit version 1.1. Available: http://java.sun.com/products/jdk/1.1/index.html. Tesser, A. M. L. & Mendolia, M. (1995). The impact of thought on attitude extremity and attitude-behavior consistency. In: R.E. Petty & J.A. Krosnick (Eds). Attitude strength: Antecedents and consequences. Ohio State University series on attitudes and persuasion, Vol. 4. (pp. 25-41). Hillsdale, NJ: Lawrence Erlbaum Associates. Welch, N. & Krantz, J.H. (1996). The World-Wide Web as a medium for psychoacoustical demonstrations and experiments: Experience and results. Behavior Research Methods, Instruments, & Computers, 28, 192-196. |
