Goal: To operationalize your research question and build all the materials necessary for its implementation on the online platform. An as-complete-as-possible draft of your protocol is due on Thursday, October 30. The finalized plan and all materials will be due the following week.
Remember that In order to operationalize a research question, you need to come up with a theory from which you derive a set conjectures. (The theory and related conjectures should be informed by your background research.) You hypothesis is a formalize statement of one or two of these conjectures. The hypothesis is what you use as a guide to plan the experimental methods, including materials, task, and procedure. It must enable you to make a testable prediction, and thus must specify what will be measured. Ideally, it should also be followed by a scenario of the possible outcomes, and how each of these will be interpreted in relation to your question. The best experiments are often those whose findings are interesting no matter what the specific outcome is (e.g., see Levitin, 1999, “Experimental Design in Psychoacoustic Research”).
There are a few things to keep in mind that pertain to practical limitations:
1. Participants: Your population sample is likely to be small and not representative of the general population (a “convenient” rather than “random” sample). You should limit the number of variables and try to maximize the number of data points you will be collecting (e.g., using a repeated measures design, that is, a design where each condition is presented to all the participants). A convenient sample also means that you will not be able to generalize your findings to the entire population, but only to the population represented by your sample (thus, the importance of providing descriptive statistics of your sample along with the results).
2. Apparatus & Data Collection: This pilot study will be conducted online using Qualtrics. This software allows for several different formats for data collection: (1) multiple choice (single-forced or multiple responses); (2) rating scale (various formats are possible); and (3) free responses. It is possible to have more than one format, but be careful to not gather more data than you can handle. If you have only limited experience with Qualtrics, you should spend some time familiarizing yourself with it. There are various issues to consider when conducting experimental studies online, but an important one to think about from the start is low motivation (failing to complete the experiment). Thus, your experiment should not take too long to complete (25 minutes is probably the maximum time you can expect someone to spend on it) and you should make all efforts to not make it too tedious.
3. Performance Measures & Statistical Analysis: Be realistic and practical in your choice of measures and statistical methods. Most measures will require some form of processing, and unless you have very good programming skills, the road from data to measure could become very arduous. Remember that although inferential statistics have a lot of rhetorical power, descriptive statistics can also be very informative (e.g., see Windsor, 2004, “Data Collection, Experimental Design,” in Empirical Musicology).
1. Make a list of all the components of your experiment (this is a preview of the “methods” section). The final list should be detailed enough to serve as a recipe for building the components. All group members should review and edit the list as necessary by the deadline.
The list will include all necessary information pertaining to:
- Task & procedure: what you will ask the participants to do and how the stimuli will be presented (ordering)
- Stimuli: what are the source materials, where they can be found, and how they will be manipulated (i.e., independent variables), including all necessary tools (hardware, software, web tools, etc.); make sure to be in contact with Michael Lorello (firstname.lastname@example.org) throughout this process!
- Data collection & analysis: what kind of data will be collected, what will be measured and how, how the data will be analyzed (only descriptive statistics are required, but you may also use correlational and inferential statistics, if you know how. NOTE: We are scheduled for a statistics workshop on Tuesday, October 28 @ CSSSI; for additional reference, see Cozby (2011), “Understanding Research Results”.
- Predicted outcome: What kind of results can you expect? Formulate how the different possible results would be interpreted in relation to your question.
2. Survey a few online experiments. How well did the design fit the experimental goal (may be presented only after the experiment is completed)? What may be some of the limitations of the methods use?
- On the “Resources” page of this site, there are a few online experiments, some of which are still active.
- You should still be able to access last semester’s group experiments on the MARL platform created by Mary Farbhood (NYU).
3. Build the experimental materials: Divide the “building” tasks between your group members. As you work on the materials, you should put samples up on the group site to be reviewed by your colleagues. We will use some of the class time on October 30 to review experimental materials and do troubleshooting; make sure to have some sample materials ready, including a few sample questions on Qualtrics.
Here is a sample list of the information that may be necessary (based on the first experiment on the MARL platform “Spontaneous grouping”):
- Study title: you will need to take into account how the title might impact your participants responses (i.e., how much do you want your participants to know about the purpose of the experiment before they complete it?).
- Study description: a concise, one sentence description that says what your study is about.
- Background: one paragraph that explains the context of your study (you may or may not want to be too specific, again depending on whether you think this might impact you participants responses).
- General instructions: a concise preview of what your participants will be doing, e.g., “You will be asked to listen to sound clips and indicate how the sounds are grouped (multiple choice response).” You may add any other special instructions about how you want your participants to go through the experiment, e.g., do you want your participants to try to do it without taking a break? If there is more than one block of trials, do you want them to take a break between them? Do you want to ask them to do the experiment in a quiet space? (NOTE: There is a standard piece of technical instructions already prepared that reads “Once you have started the experiment, please do not use the “Back” browser button or refresh the page. By doing so you will interrupt the experiment and it will result in technical problems.”)
- Part Description (Part 1 of #): This can be the same as the instructions in the previous page or a more/less detailed version (if your study has more than one part, you will need to provide one for each).
- Trial Instructions (Part #): specific instructions for participants’ task, including how they will respond, e.g., “[Above play button:] Click on the play button to hear the sound. After the sound has finished playing, select a response. Press the continue button to continue to the next trial. [Below play button and above response boxes and continue button:] “How many tones per group did you hear?” You should also provide the response format, e.g., multiple-choice questions: 1, 2, 3, 4; Yes/No; happy/sad/calming/exciting. (NOTE: If your study has more than one part you will need to give similar information for each part.)
- End (of Part #): You may thank your participants and/or given them some more information about the purpose of your study, how to contact you if they want to see the results when the study is completed, etc. (If your study is in many parts, you may instruct them to take a break at this point or to continue on.)
NOTE: We will work on the background survey and post-experiment questionnaire together. You may want to draft some of the questions you would like to have included.