W4: Effective Web Instruction, Part II

To continue reviewing Ted Frick and Elizabeth Boling's Effective Web Instruction: Handbook for Inquiry-Based Process, this summary will cover the second two parts: Preparing for Testing a Prototype and Testing a Prototype.

When testing a prototype, a designer should address 3 main questions:

1. Is the instruction effective?
2. Are the students satisfied with the instruction?
3. Is the product usable by students?

In order to address the above, a designer should start by developing a paper prototype. This is supposed to save time on the development end and also encourage testers to be more honest about their feedback, since the idea is that the more draft a prototype looks, the more critical they should be willing to be if something doesn't seem right to them.
(Personally, I feel like the directions for developing the paper prototype sound pretty darn time consuming. At first I thought based on Myer's description of it that it would be a fast mock up, kind of like wireframing we did for Infographics in 541, but by reading what Frick and Boling want for a paper prototype it is incredibly detailed! With "links" that actually link to tabbed pages, etc. I haven't ever done one before, but I feel like I'm going to be spending about as much time making a "simple" word document prototype as I would an HTML version...just without pictures.)

Before administering the prototype to an authentic tester, one should administer a pre-mastery assessment, to make sure that the students don't already have all the skills needed to "pass" the mastery assessment.
(I'm concerned about the time aspect of this... my master assessment will include creating a mini-lesson based on an authentic text. So how much time do I have them work on creating this mini-lesson? For the assessment, it will probably be about 60 min. These are volunteers, so I'm always nervous about how much time I'm expecting them to donate to me.)

Before having the tester test the prototype, the observer should give the tester short and simple explanations of how the prototype works. For example, how to "click" the links." And they should ask the tester to *think* aloud as the work, and point out any problems they have. If during the actual process the tester isn't thinking aloud enough and/or not giving enough information with too general thoughts, the observer should prompt the tester. However, the observer should not answer questions or give help...just record what they observe as the tester works through the prototype.
(Would it be okay to start with a quick example prototype and demonstrate just what kind of thinking aloud we expect? I think that would be helpful to show them what kind of reasoning and thoughts we are really looking for.)

After the tester finishes the prototype, the observer should administer the mastery assessment. If the tester cannot successfully complete the mastery assessment, the designers must figure out why not and look for problems. However, Frick and Boling that even if they do complete the mastery assessment correctly, that does not mean there are no problems either.

At this point, the observer should also administer a formative evaluation survey (using the Likert scale) asking the tester to rate how they felt about the prototype.

Finally, the designers (and observers) should gather and review the data from the observation/prototype testing. They should be specifically interested in looking for patterns of problems. Based on this data and the results of the formative survey, they can decide what changes to make to the instruction/design.
(According to Frick and Boling, at this point we should make changes and re-test, and keep testing until the sample is saturated -no more patterns can be found; however, for the sake of time in this class, I do not see that as a logistical possibility. )