Home » 2014 » February

Monthly Archives: February 2014

Cohorts in self-paced environments: do MOOCs make it easier

On Friday, I had a conversation with a colleague who asked how you create cohorts in a self-paced course/program. After all, in most courses you have a group of students theoretically all at the same place in the content, who should be able to carry on discussions and debates. Such a setup allows for the creation of a community of learners within the course structure–one of the  circles in the Community of Inquiry Model.  But, the traditional cohort (or should we say: default cohort model) is not easy, sometimes not even possible, to achieve in a competency-based, self-paced or personalized learning environment.

The easiest method is to have a large enough base of students that you can group students according to where they are in the content. It would function like the gaming sites that match you with awaiting opponents. The group can be as little as two students, who work together throughout the course or, if the course is not linear, two students who happen to enter the “game room” (or discussion forum) at approximately the same time. The LMS could be set up to notify students when someone has entered and posted a comment, for example…most do this now, so the student expectations of how this will work would be the only thing that would need some tweaking.

But, students can also create their own cohorts. The meetups that have developed around some of the MOOCs are one example of this. Cohorts are not the only way of creating a base of individuals that the student can interact with, and gain insight from. In reality, social networks can replace the cohort model. With a bit of research and/or guidance, there are many places in the authentic world where students could get feedback on ideas, thoughts, assignments, etc. and interact with individuals around a given topic. The benefit is that it can increase their professional and personal network.

There are many places where students can engage in a debate on a topic, post a comment, or submit their research for scrutiny. One site in particular comes to mind: PLOS ONE: http://www.plosone.org/ and all of the sister sites. Today, I found an article on PLOS Biology entitled “Right Brain Left Brain: Fact and Fantasies“. It was published on January 21st (less than a month ago). Below are the statistics for this article. Note that there is an area for comments. And because it is published using an Open Access license, others can build and contribute to this research easily.

PLOSBiology data on article: 7801 views, 917 PDF downloads

Manufactured cohorts are another option. These take creativity and time on the part of the ID and/or SME to create. They will not work in a project-based environment, but are possible in discussion forums. Students can make posts and respond to posts.

A bit of creative thinking can create a highly effective and engaging course without the use of a traditional cohort. In many cases the traditional cohort model was not that effective anyway. Students were only theoretically at the same point in the course, and points are often given for less than optimal participation.  The construction is really meant to serve schools, as it allows for easier grading (easier than assessing social networking assignments), easier construction of courses, and quicker ways of assessing the effectiveness of the given content as everything is normed. What all of this is, however, is not authentic, not experiential, and though often designed to use constructivist methodologies, its effect is often less that ideal–time is a factor that significantly impacts the ability of students to construct knowledge.


Types of Assessments–to get you thinking

Generally speaking there are two types of assessments in learning: formative and summative. Formative assessments provide useful feedback to the student and are used for the purposes of increasing understanding. Summative assessments measure achievement. In traditional courses, students are given points for formative assessments and summative assessments. However, in the case of true outcomes based education, especially  CBL,  formative assessments should not receive points as they not measure mastery, but help build mastery. Formative assessments involve self-assessment and/or practice of competencies.

Let’s look at specific activities and assessments to see how this works in true outcomes based learning (competency based).

Q: Can a test be used to measure a competency?
A: Yes, with caveats. It cannot be an open book test, must have a time limit. Students should not be able to change answers, or be allowed multiple attempts if the test is measuring a competency. It is best to have each question on it’s own page and not allow backtracking. They should be built using ell-constructed questions to measure competency.

Q: What kind of competencies can a test measure?
A: Tests are particularly good for measuring knowledge, or being able to define terms. For example if a competency states: Student will be able to identify the parts of a cell and their function.

Q: When do points apply to tests?
A: Test and quizzes can be wonderful tools for self-evaluations, practice, and even for teaching content especially if they include comprehensive feedback, and (in the case of practice and teaching) allow for multiple attempts. However, these formative assessments should not have points associated with them. When tests are used to measure a competency, they should have points associated with them. The points should be an indicator of the weight and level of the competency it is measuring. Generally, there should be an assignment somewhere in the course that measures the application of the knowledge the test is measuring. Since that assessment will measure both the knowledge and the ability to apply that knowledge, it should have a greater weight.

Discussion Forums:

Q: Should discussion forums be eliminated from a CBL course
A: That depends. The problem with the construction of most discussion forums is that they require at least a small cohort of students who can take part in the discussion. Depending on the construction of a forum, it can be used to measure a competency (summative) or to develop deeper understanding (learning activity or formative assessment).

Q: How can a forum be used to measure competency?
A: Forums (or any social networking activity) requires a great deal of thought in design, and generally a great deal of time and effort on the part of faculty in monitoring the discussion. Here is an example of a competency that might be measured using a forum: “Student demonstrates the ability to debate and argue a case…” The difficulty comes when the courses are designed for self-paced learning. How does one debate if there is no one to debate with? We tackle that in the next question

Q: Can a forum be used in a self-paced course?
A: Yes, with a great deal of planning and assistance from technology. For example, if the technology can work like gaming rooms, a student can enter the forum and “wait” for another student (or group of students) to “enter” the room. This would require a course that is not completely linear, so students can go back to that module when there are enough students in the forum to make it work efficiently.

Research Papers:
Q: Can a research paper be used to demonstrate competency?
A: Yes, but again that depends on how the assignment is constructed and the competency it is measuring. In general they are a learning activity–the student dives deeply into a particular subject and learns about it. But if there is a competency for demonstrating the ability to find and cite sources then a research paper then the that assignment would be good method for assessing the competency. However, generally speaking, the assignment should also include a presentation of the research with an extensive Q & A session (defend their conclusions)—to insure the student has mastered the material, and not merely regurgitated material.

Q: Should we assign points for homework?

A: Homework should never be used to measure an outcome/competency. Homework does not measure mastery. It is practice and therefore should not have points associated with it. Homework should, however, receive feedback or be followed by a practice quiz that provides feedback. (test for understanding)

Projects, Scenarios, Simulations:
Q: We use these for learning activities, but can they also be used for measuring mastery?
A: Yes, projects, scenarios, and simulations can provide for authentic (or close to authentic) summative assessments. With scenarios and simulations, there should be a small margin or error for demonstrating competency (or mastery). A scenario or simulation that was used for training purposes, should not be reused, as is, for assessment of mastery. They should be changed somewhat. Also, while there might’ve been hints or other feedback provided during the training, hints should not be allowed in the summative assessment. Projects should be constructed in such a way as to mimic one that a student would need to complete in the “real” world, with the same or similar expectations.

Interviews & Observations: live, video and audio:
Q: What other methods can be used for assessing competency?
A: Interviews are excellent methods for assessing mastery level learning, especially if the student is not given the questions beforehand and is not allowed to use notes. Observing the student, particularly in clinical/classroom/workplace environments are also excellent ways of measuring competency.

Q: Can observations also be a useful tool for helping students gain competency?
A: Yes, observations can be an excellent tool for learning. For example, when an athlete or performer watches a video of themselves, they can observe where they need improvement and/or what is working well for them. It is an excellent tool for self-assessment. For example, having students record themselves giving a speech, allows them to hear how many “ums” they say, as well as their cadence. When others use an observation to provide constructive feedback, it can also be an excellent learning tool.

Blogs and other forms of Journaling:

Q: What about blogs, reflections and personal journals–those are mostly for processing and learning, right?
A: Journals and blogs can be excellent tools for measuring certain types of competencies, especially when they are accompanied by an artifact of learning. For example, if a student in a language acquisition program creates an audio recording of themselves speaking in the language, and accompanies that with a reflection that includes: what they are saying, why they chose that, how it is expressed culturally, etc, you get a much better understanding of the student’s grasp of the language. Reflections and blog posts can also be powerful tools for assessing mastery gained in internships. For example, in a blog post the student can discuss their experiences and what they are learning. Tests can no be designed to measure somethings that only blog posts and personal reflections can reveal.

I hope you found this helpful. Feel free to post your suggestions in the comment section.

Curriculum vs Instruction Design

empty slate (chalkboard)The past few posts have focused on Competency-Based Education, which is generally speaking a Curriculum Design Model. Designing curriculum to follow a specific model, does require instructional methods that support that design, so curriculum designs can also be instructional design models. But this is not necessarily true in reverse: instructional design models can exist independently from curriculum designs. For example, Inquiry Based Models are a type of instructional design that do not require a specific type of curriculum model.

When an institution undertakes a curriculum redesign, this is usually done by a team of individuals mostly built of faculty. Curriculum redesigns are both labor and time intensive, require extensive buy-in, and almost always require multiple levels of approval. It many times requires the approval of an accrediting body as well.  Curriculum redesigns can benefit from a curriculum specialist, who can assist with mapping and other strategic exercises. The redesign teams can also benefit from the wealth of knowledge and experience a curriculum specialist can bring to the table with regards to models and what other institutions have found effective, not effective, or worthy of more work.

Instructional design, on the other hand, is concerned with the teaching in a particular course. With faculty involved as subject matter experts, and in choosing the method of instruction (problem-based, inquiry-based, etc), the course development should also include an instructional designer. There is some confusion about the roles of IDs and often resistance from faculty in using an ID. I might even go so far as to say that some faculty even resent the “intrusion” of IDs. There are two common claims faculty make when dismissing the need for an ID: “I’m the subject matter expert” (meaning: “they don’t know how to  teach my course; I do”), and “…academic freedom…” which is a much often misunderstood and overused term, and has nothing to do with effective instruction.  When it comes to how people learn best and how to design effective instruction, Instructional Designers, and other specialists in education and assessment strategies, are the instructional “Subject Matter Experts.” IDs can help with the construction of a project, help with creative ideas, help with constructing effective assessments, help with constructing effective discussion questions and more. Instructional Designers do not tell faculty what to teach, nor even how to teach it, but they can help with constructing these things to be far more effective. It’s what they studied and what they continue to study every single day.


Converting traditional formats to CBE

Today I’m going to share with you a simplified format for converting a traditional online (and possibly f2f) course to a competency based model. What I’d like you to notice is that the development focus is on assessment, not content. What I’d like you to keep in mind is that feedback, assessments and adaptive learning scenarios do not have to be computer mediated; they can be provided by instructors.

I’d like to begin by sharing with you and image I created that explains the basic premise of a competency based course. That is that progress is not linear, and that students do not progress to the final assessment until they are successful in the formative assessments (which they can repeat as often as needed).

Diagram of CBE shows articulate competencies to content to practice to assessment and then a loop either back to content  practice or out to final assessment


Well thought out feedback is essential, as is the timeliness of the feedback, so that students know where they stand and can take action to remedy weaknesses. These formative assessments should provide a more comprehensive feedback than “correct” or “incorrect”. In fact, they need to be prescriptive. The LMS can be set up to automatically lead students, based on their formative assessment, to remediation or to the final assessment (if they are ready). Or, this can be done through council with an instructor, who goes over the results with the student and provides a recommended path to success. Or, it can be a combination of the two.

Now, let’s look at the steps a designer and subject matter expert should consider when converting an existing course to competency based. Please note: this is not meant to be a development roadmap, but rather a guideline for considerations. It is not meant to be highly detailed.

Step 1: assess current course objectives to insure they are stated in terms of competencies—what students must demonstrate in terms of the application of knowledge and skills to complex tasks.

Step 2: sort course objectives/competencies into progressive milestones or module steps that will lead to the final assessment

Step 3*: assign current assessments and activities to applicable milestones where appropriate.

Step 4*: determine whether any of the current assessments and activities are more suited for practice and self-assessments than for a summative evaluation of competency? For those that are, change the points allocated, if necessary, to reflect the formative nature of the assessment, and insure proper formative feedback is provided

Step 5*: identify and build summative competency-based assessment(s) including rubric(s)

Step 6: identify and build (where necessary) practices and self-assessments insuring students are given the opportunity to practice all outcomes and receive feedback that clearly tells student if he/she is ready to progress to the summative evaluation of competency, and if not what steps the students need to take in order to meet the acceptable level of competency.

Step 7: Build in additional content for remedial purposes.

Step 8: review all instructive course materials, mapping to Milestones, practices and assessments. Identify any gaps or unnecessary redundancies.

Step 9: Review all rubrics for point matching to levels and areas of competency achieved

*Steps 3, 4, and 5 are not is a specific order, should be looped until all the developers of the course are satisfied with the construction and point allocation for each assessment

What is important to notice is that a CBE course will require more content, will require practices, and will require thoughtfully constructed assessments and point systems. What you do not want to do is to provide a path for a student to pass a course without having met the necessary competencies.