Generally speaking there are two types of assessments in learning: formative and summative. Formative assessments provide useful feedback to the student and are used for the purposes of increasing understanding. Summative assessments measure achievement. In traditional courses, students are given points for formative assessments and summative assessments. However, in the case of true outcomes based education, especially CBL, formative assessments should not receive points as they not measure mastery, but help build mastery. Formative assessments involve self-assessment and/or practice of competencies.
Let’s look at specific activities and assessments to see how this works in true outcomes based learning (competency based).
Q: Can a test be used to measure a competency?
A: Yes, with caveats. It cannot be an open book test, must have a time limit. Students should not be able to change answers, or be allowed multiple attempts if the test is measuring a competency. It is best to have each question on it’s own page and not allow backtracking. They should be built using ell-constructed questions to measure competency.
Q: What kind of competencies can a test measure?
A: Tests are particularly good for measuring knowledge, or being able to define terms. For example if a competency states: Student will be able to identify the parts of a cell and their function.
Q: When do points apply to tests?
A: Test and quizzes can be wonderful tools for self-evaluations, practice, and even for teaching content especially if they include comprehensive feedback, and (in the case of practice and teaching) allow for multiple attempts. However, these formative assessments should not have points associated with them. When tests are used to measure a competency, they should have points associated with them. The points should be an indicator of the weight and level of the competency it is measuring. Generally, there should be an assignment somewhere in the course that measures the application of the knowledge the test is measuring. Since that assessment will measure both the knowledge and the ability to apply that knowledge, it should have a greater weight.
Q: Should discussion forums be eliminated from a CBL course
A: That depends. The problem with the construction of most discussion forums is that they require at least a small cohort of students who can take part in the discussion. Depending on the construction of a forum, it can be used to measure a competency (summative) or to develop deeper understanding (learning activity or formative assessment).
Q: How can a forum be used to measure competency?
A: Forums (or any social networking activity) requires a great deal of thought in design, and generally a great deal of time and effort on the part of faculty in monitoring the discussion. Here is an example of a competency that might be measured using a forum: “Student demonstrates the ability to debate and argue a case…” The difficulty comes when the courses are designed for self-paced learning. How does one debate if there is no one to debate with? We tackle that in the next question
Q: Can a forum be used in a self-paced course?
A: Yes, with a great deal of planning and assistance from technology. For example, if the technology can work like gaming rooms, a student can enter the forum and “wait” for another student (or group of students) to “enter” the room. This would require a course that is not completely linear, so students can go back to that module when there are enough students in the forum to make it work efficiently.
Q: Can a research paper be used to demonstrate competency?
A: Yes, but again that depends on how the assignment is constructed and the competency it is measuring. In general they are a learning activity–the student dives deeply into a particular subject and learns about it. But if there is a competency for demonstrating the ability to find and cite sources then a research paper then the that assignment would be good method for assessing the competency. However, generally speaking, the assignment should also include a presentation of the research with an extensive Q & A session (defend their conclusions)—to insure the student has mastered the material, and not merely regurgitated material.
Q: Should we assign points for homework?
A: Homework should never be used to measure an outcome/competency. Homework does not measure mastery. It is practice and therefore should not have points associated with it. Homework should, however, receive feedback or be followed by a practice quiz that provides feedback. (test for understanding)
Projects, Scenarios, Simulations:
Q: We use these for learning activities, but can they also be used for measuring mastery?
A: Yes, projects, scenarios, and simulations can provide for authentic (or close to authentic) summative assessments. With scenarios and simulations, there should be a small margin or error for demonstrating competency (or mastery). A scenario or simulation that was used for training purposes, should not be reused, as is, for assessment of mastery. They should be changed somewhat. Also, while there might’ve been hints or other feedback provided during the training, hints should not be allowed in the summative assessment. Projects should be constructed in such a way as to mimic one that a student would need to complete in the “real” world, with the same or similar expectations.
Interviews & Observations: live, video and audio:
Q: What other methods can be used for assessing competency?
A: Interviews are excellent methods for assessing mastery level learning, especially if the student is not given the questions beforehand and is not allowed to use notes. Observing the student, particularly in clinical/classroom/workplace environments are also excellent ways of measuring competency.
Q: Can observations also be a useful tool for helping students gain competency?
A: Yes, observations can be an excellent tool for learning. For example, when an athlete or performer watches a video of themselves, they can observe where they need improvement and/or what is working well for them. It is an excellent tool for self-assessment. For example, having students record themselves giving a speech, allows them to hear how many “ums” they say, as well as their cadence. When others use an observation to provide constructive feedback, it can also be an excellent learning tool.
Blogs and other forms of Journaling:
Q: What about blogs, reflections and personal journals–those are mostly for processing and learning, right?
A: Journals and blogs can be excellent tools for measuring certain types of competencies, especially when they are accompanied by an artifact of learning. For example, if a student in a language acquisition program creates an audio recording of themselves speaking in the language, and accompanies that with a reflection that includes: what they are saying, why they chose that, how it is expressed culturally, etc, you get a much better understanding of the student’s grasp of the language. Reflections and blog posts can also be powerful tools for assessing mastery gained in internships. For example, in a blog post the student can discuss their experiences and what they are learning. Tests can no be designed to measure somethings that only blog posts and personal reflections can reveal.
I hope you found this helpful. Feel free to post your suggestions in the comment section.
The past few posts have focused on Competency-Based Education, which is generally speaking a Curriculum Design Model. Designing curriculum to follow a specific model, does require instructional methods that support that design, so curriculum designs can also be instructional design models. But this is not necessarily true in reverse: instructional design models can exist independently from curriculum designs. For example, Inquiry Based Models are a type of instructional design that do not require a specific type of curriculum model.
When an institution undertakes a curriculum redesign, this is usually done by a team of individuals mostly built of faculty. Curriculum redesigns are both labor and time intensive, require extensive buy-in, and almost always require multiple levels of approval. It many times requires the approval of an accrediting body as well. Curriculum redesigns can benefit from a curriculum specialist, who can assist with mapping and other strategic exercises. The redesign teams can also benefit from the wealth of knowledge and experience a curriculum specialist can bring to the table with regards to models and what other institutions have found effective, not effective, or worthy of more work.
Instructional design, on the other hand, is concerned with the teaching in a particular course. With faculty involved as subject matter experts, and in choosing the method of instruction (problem-based, inquiry-based, etc), the course development should also include an instructional designer. There is some confusion about the roles of IDs and often resistance from faculty in using an ID. I might even go so far as to say that some faculty even resent the “intrusion” of IDs. There are two common claims faculty make when dismissing the need for an ID: “I’m the subject matter expert” (meaning: “they don’t know how to teach my course; I do”), and “…academic freedom…” which is a much often misunderstood and overused term, and has nothing to do with effective instruction. When it comes to how people learn best and how to design effective instruction, Instructional Designers, and other specialists in education and assessment strategies, are the instructional “Subject Matter Experts.” IDs can help with the construction of a project, help with creative ideas, help with constructing effective assessments, help with constructing effective discussion questions and more. Instructional Designers do not tell faculty what to teach, nor even how to teach it, but they can help with constructing these things to be far more effective. It’s what they studied and what they continue to study every single day.
Today I’m going to share with you a simplified format for converting a traditional online (and possibly f2f) course to a competency based model. What I’d like you to notice is that the development focus is on assessment, not content. What I’d like you to keep in mind is that feedback, assessments and adaptive learning scenarios do not have to be computer mediated; they can be provided by instructors.
I’d like to begin by sharing with you and image I created that explains the basic premise of a competency based course. That is that progress is not linear, and that students do not progress to the final assessment until they are successful in the formative assessments (which they can repeat as often as needed).
Well thought out feedback is essential, as is the timeliness of the feedback, so that students know where they stand and can take action to remedy weaknesses. These formative assessments should provide a more comprehensive feedback than “correct” or “incorrect”. In fact, they need to be prescriptive. The LMS can be set up to automatically lead students, based on their formative assessment, to remediation or to the final assessment (if they are ready). Or, this can be done through council with an instructor, who goes over the results with the student and provides a recommended path to success. Or, it can be a combination of the two.
Now, let’s look at the steps a designer and subject matter expert should consider when converting an existing course to competency based. Please note: this is not meant to be a development roadmap, but rather a guideline for considerations. It is not meant to be highly detailed.
Step 1: assess current course objectives to insure they are stated in terms of competencies—what students must demonstrate in terms of the application of knowledge and skills to complex tasks.
Step 2: sort course objectives/competencies into progressive milestones or module steps that will lead to the final assessment
Step 3*: assign current assessments and activities to applicable milestones where appropriate.
Step 4*: determine whether any of the current assessments and activities are more suited for practice and self-assessments than for a summative evaluation of competency? For those that are, change the points allocated, if necessary, to reflect the formative nature of the assessment, and insure proper formative feedback is provided
Step 5*: identify and build summative competency-based assessment(s) including rubric(s)
Step 6: identify and build (where necessary) practices and self-assessments insuring students are given the opportunity to practice all outcomes and receive feedback that clearly tells student if he/she is ready to progress to the summative evaluation of competency, and if not what steps the students need to take in order to meet the acceptable level of competency.
Step 7: Build in additional content for remedial purposes.
Step 8: review all instructive course materials, mapping to Milestones, practices and assessments. Identify any gaps or unnecessary redundancies.
Step 9: Review all rubrics for point matching to levels and areas of competency achieved
*Steps 3, 4, and 5 are not is a specific order, should be looped until all the developers of the course are satisfied with the construction and point allocation for each assessment
What is important to notice is that a CBE course will require more content, will require practices, and will require thoughtfully constructed assessments and point systems. What you do not want to do is to provide a path for a student to pass a course without having met the necessary competencies.
In 2010 the journal Medical Teacher devoted an entire issue to competency based medical education: http://informahealthcare.com/toc/mte/32/8. This issue is well worth reading by any educator in higher education that is interested in competency based education. While the articles are written about medical education, which many in the liberal arts reject as “practical” rather than “theoretical”, and therefore not applicable to undergraduate education, those with more insight will learn how to develop stronger competency based programs, and the value of a competency based program.
Today’s Inside Higher Ed, featured yet another article on the move to competency based education: http://www.insidehighered.com/quicktakes/2014/01/23/colleges-pitch-possible-experiments-competency-based-programs. It’s a short article stating that a number of colleges and university are preparing to launch competency based programs. But the list of participating institutions is not one that contains any surprises. These are colleges and universities that are known for non-traditional curriculum. What might surprise some is the list of medical schools moving to competency based programs, and that the Carnegie Foundation for the Advancement of Teaching, in its 100 year anniversary of the much respected Flexnor Report, advocated for competency based medical education. In it’s 2010 report Educating Physicians: A Call for Reform of Medical School and Residency, the authors’ recommended that medical schools “standardize learning outcomes and assess competencies over time.” and added: “A focus on learning outcomes and milestones could end the time-based structure of medical school and residency.”
Many medical schools have taken the advice of this report quite seriously, and have been working on converting existing curriculum to a competency based format. The rest of us could do well to learn from what they are doing. It may not all be applicable, but there is much to be learned from their thoughtfulness and efforts.
I highly recommend that those who are interested in Competency Based Education in Higher Education read Medical Teacher’s issue 8 of volume 32.