Summary of 5 Star Instructional Design Rating by M. David Merrill
Merrill's 5-Star Instructional Design Rating is an evaluation tool for rating courseware according to the First Principles of Instructional Design, using a three star rating method (bronze, silver, and gold).
For each of the first principles identified, task-centered approach, activation, demonstration, application, and integration (Merrill et al, 2008), Merrill provides three questions to be used for evaluating how well the principle is being addressed within the instruction. It is not clearly stated, however, one can deduce that if all questions are "yes" than that principle receives a gold star rating. If two of three questions are "yes" than the principle receives a silver star. And of course, if only one question is "yes" than it receives a Bronze. Presumably, if all questions are "no", then no star is awarded.
My biggest issue with this rating system is that it's too subjective. I would think it is a fine tool for Merrill to use, because he knows exactly what his own criteria is for the questions. However, I don't think if you tested it for reliability between a set of different raters that there would be a consistent rating between the raters.
For example: 3. b. (2) Multiple representations are used for the demonstrations? Are two representations sufficient? Or three? Or four? What constitutes multiple? Or 3. c. Is media relevant to the content and used to enhance learning. What would be considered relevant to one evaluator might not be equally relevant to a rater who has higher standards/more experience/different expectations to another rater.
So while it seems like a good start, I really feel that if this is going to be tool used universally for evaluating instruction, more work needs to go into developing descriptive parameters to improve reliability between raters. (I haven't checked my peers' evaluations yet, but if two of us evaluated the same instruction, it would be interesting to comparing our ratings in order to examine the 5-star rating reliability.)
Speaking of reviewing: most computer mediated learning that I'm familiar with fall into the receptive ("spray and pray") or exploratory ("sink-or-swim") categories, so I decided to evaluate one of the examples that Dr. Myers provided us. I chose the Understanding Creditor Statements by Wendy Baez (2006).
| Is the instructional architecture tutorial or experiential? | Yes. The course is a tutorial for understanding credit card terms, statements, and finance charges. I would say it does go beyond receptive or exploratory. | |||
| Is the courseware TELL-&-ASK (T&A) instruction? | Here is where I'm on the fence. Yes, this courseware provides multiple choice/short answer questions as part of the individual modules (with immediate feedback), which seems to fit the description of Tell&Ask. And, it also has a final assessment, which would seem to not be T&A. But then again, the final assessment seems to also just be a multiple choice test. So how is that different than being T&A? The learner doesn't have to produce anything as part of demonstrating understanding. I'm going to assume, since this was provided as a sample to review that Dr. Myers does not consider this T&A, thus making it inappropriate for this 5-star rating. However, I think I will need more discussion to understand why these questions are not considered just Tell&Ask. |
|||
| a. Does the courseware show learners the task they will be able to do or the problem they will be able to solve as a result of completing a module or course? | b. Are students engaged at the problem or task level not just the operation or action levels? | c. Does the courseware involve a progression of problems rather than a single problem? | ||
| 1. Is the courseware presented in the context of real world problems? | No. The modules clearly states the learning objects and sometimes uses examples as part of the lesson. However, they do not begin the instruction with a demonstration/authentic example of what the learner will be able to do as a result of the training. | Yes. I find the practice engaging. It could be improved by asking learners to practice on their own bank statements (for example for Lesson 2: Locating Statement Information especially), however there are logistical reasons why that probably wouldn't work a t this time. | Yes. Each module has questions and/or practice sections. And then there is a final assessment at the very end of the course. |  |
| a. Does the courseware direct learners to recall, relate, describe, or apply knowledge from relevant past experience that can be used as a foundation for new knowledge? | b. Does the courseware provide relevant experience that can be used as a foundation for the new knowledge? | c. If learners already know some of the content are they given an opportunity to demonstrate their knowledge? | ||
| 2. Does the courseware attempt to activate relevant prior knowledge or experience? | Yes. The very first thing the course does in Lesson 1 is show a quote and say "Do you recognize the previous statement?" And throughout the course, there is reference to the learners experience and own personal statements. | Yes. I feel like the use of authentic statements and calculators helps build a firm foundation for applying the new knowledge. | No. I do not see where the learner can demonstrate prior knowledge. The course seems to focus on the new information being taught. | |
| a. Are the demonstrations (examples) consistent with the content being taught? (1) Examples and non-examples for concepts? (2) Demonstrations for procedures? (3) Visualizations for processes? (4) Modeling for behavior? | b. Are at least some of the following learner guidance techniques employed? (1) Learners are directed to relevant information? (2) Multiple representations are used for the demonstrations? (3) Multiple demonstrations are explicitly compared? | c. Is media relevant to the content and used to enhance learning? | ||
| 3. Does the courseware demonstrate (show examples) of what is to be learned rather than merely tell information about what is to be learned? | Yes. The examples used (interactive statements, statements, calculators, case studies, etc) and good examples and useful for the learner to understand the concepts and visualizations. | Yes and no. There are multiple representations (examples) used, however I don't feel they are explicitly compared. And there really isn't any redirecting to relevant information (unless you count linking the the glossary page as redirecting to relevant information). | This really depends on one's definition of media. The course does not use music or video, but it does use pictures. And it even has a picture with roll over features on it for a more interactive experience. So, I guess I will give it a Yes, mostly due to the interactive statement. |  |
| a. Are the application (practice) and the posttest consistent with the stated or implied objectives? (1) Information-about practice requires learners to recall or recognize information. (2) Parts-of practice requires the learners to locate, name, and/or describe each part. (3) Kinds-of practice requires learners to identify new examples of each kind. (4) How-to practice requires learners to do the procedure. (5) What-happens practice requires learners to predict a consequence of a process given conditions, or to find faulted conditions given an unexpected consequence. | b. Does the courseware require learners to use new knowledge or skill to solve a varied sequence of problems and do learners receive corrective feedback on their performance? | c. In most application or practice activities, are learners able to access context sensitive help or guidance when having difficulty with the instructional materials? Is this coaching gradually diminished as the instruction progresses? | ||
| 4. Do learners have an opportunity to practice and apply their newly acquired knowledge or skill? | Yes. Module 2 and Module 3 especially have good practice sections that use different kinds of practice to support the state objectives and require the leaner to demonstrate understanding in order to answer the questions correctly. | Yes. Learners have to solve a variety of problems and are provided immediate feedback. | Yes and no. There is no "Help" section or place for the learner to directly go for help. However, new terms are hyperlinked throughout the course, linking back to definitions and/or explanations. | |
| a. Does the courseware provide an opportunity for learners to publicly demonstrate their new knowledge or skill? | b. Does the courseware provide an opportunity for learners to reflect-on, discuss, and defend their new knowledge or skill? | c. Does the courseware provide an opportunity for learners to create, invent, or explore new and new knowledge or skill? | ||
| 5. Does the courseware provide techniques that encourage learners to integrate (transfer) the new knowledge or skill into their everyday life? | No. There is an assessment the learner can take at the end. It is through a 3rd party website (but the link is not active). However, I wouldn't consider this publicly demonstrating knowledge. | No. The questions and practice do not seem to lead themselves for discussion, reflection, or collaboration. | No. | No star. |
| Overall rating: | |
|||
REFERENCE
Merrill, M. D., Barclay, M., & Schaak, A. v. (2008). Prescriptive Principles for Instructional Design. In AECT Handbook (pp. 173-184).
No comments:
Post a Comment