Home » Online Learning
Category Archives: Online Learning
Few people would publish a book or paper without having it first reviewed by an editor, or even simply someone they trust to have a good handle on the language. Yet, most colleges and universities publish online courses without a once over by an editor. I realize that some of this is due to an incorrect interpretation of academic freedom, but I’m not going to discuss that in this article. The consequence of errors in a public publication falls primarily on the author (or journal), however the consequence of errors in an online course primarily falls on the recipient (the student).
My experience, in the field of course design and development, is that while schools build teams of instructional designers, and invest in multimedia programs to enhance their online courses, few have course editors on their teams. And, while I would never claim to be capable of being an editor myself, my experience as a student in online courses, and as a course reviewer, is that many courses have significant errors in grammar, spelling, and syntax. In one of my graduate courses, the language was so bad it became a huge distraction. While I had a genuine respect for this individual’s knowledge of the subject matter, it was clear the individual had great difficulty in communicating clearly in writing. The large number of errors made some parts of the content almost unreadable, and the descriptions of various assignments extremely unclear. This problem existed throughout the entire semester. A course editor could’ve made all of this better for everyone concerned: the professor and the students. This is by far not the only course I’ve seen with these issues, though this was probably the worst.
Recently, I experienced this same problem in a MOOC offered by Stanford. The course had probably spent months in development. It was nicely designed, with high-end and entertaining video. But, there were numerous blatant grammatical errors in the audio, and text. A large portion of the forum posts complained of the language errors–some quite strongly. What made this particularly problematic, was that many of the students were not native English speakers.
What are the negative consequences of poorly written courses? Well, the first is that credibility can be negatively impacted. Cognitively, it increases the effort a student needs to process the information–sometimes significantly. In terms of Cognitive Load Theory, it increases extraneous load–which means it is more difficult for the student to learn the material–it negatively impacts learning. But, it can also give students the incorrect information, and it can create problems with assessment of student learning. For example, if a student misunderstands a poorly written question, or misunderstands what the assessment is asking the student to produce, then the student’s answer is an inaccurate measurement of what the student knows or is capable of doing. The assessment is invalid.
Often, we can use the context of a sentence to help us correctly interpret a poorly written sentence. For example, if we read a note: “Let’s eat Mom”, we would know the sentence is not suggesting we eat Mom, but rather “Let’s eat, Mom”. However, grammatically incorrect content, with which we have little familiarity or limited understanding, may not be easily interpreted correctly.
It is important for institutions to think of online courses as published materials, and insure they are reviewed by editors to the same degree that the school would like all of their public facing materials reviewed.
Generally speaking there are two types of assessments in learning: formative and summative. Formative assessments provide useful feedback to the student and are used for the purposes of increasing understanding. Summative assessments measure achievement. In traditional courses, students are given points for formative assessments and summative assessments. However, in the case of true outcomes based education, especially CBL, formative assessments should not receive points as they not measure mastery, but help build mastery. Formative assessments involve self-assessment and/or practice of competencies.
Let’s look at specific activities and assessments to see how this works in true outcomes based learning (competency based).
Q: Can a test be used to measure a competency?
A: Yes, with caveats. It cannot be an open book test, must have a time limit. Students should not be able to change answers, or be allowed multiple attempts if the test is measuring a competency. It is best to have each question on it’s own page and not allow backtracking. They should be built using ell-constructed questions to measure competency.
Q: What kind of competencies can a test measure?
A: Tests are particularly good for measuring knowledge, or being able to define terms. For example if a competency states: Student will be able to identify the parts of a cell and their function.
Q: When do points apply to tests?
A: Test and quizzes can be wonderful tools for self-evaluations, practice, and even for teaching content especially if they include comprehensive feedback, and (in the case of practice and teaching) allow for multiple attempts. However, these formative assessments should not have points associated with them. When tests are used to measure a competency, they should have points associated with them. The points should be an indicator of the weight and level of the competency it is measuring. Generally, there should be an assignment somewhere in the course that measures the application of the knowledge the test is measuring. Since that assessment will measure both the knowledge and the ability to apply that knowledge, it should have a greater weight.
Q: Should discussion forums be eliminated from a CBL course
A: That depends. The problem with the construction of most discussion forums is that they require at least a small cohort of students who can take part in the discussion. Depending on the construction of a forum, it can be used to measure a competency (summative) or to develop deeper understanding (learning activity or formative assessment).
Q: How can a forum be used to measure competency?
A: Forums (or any social networking activity) requires a great deal of thought in design, and generally a great deal of time and effort on the part of faculty in monitoring the discussion. Here is an example of a competency that might be measured using a forum: “Student demonstrates the ability to debate and argue a case…” The difficulty comes when the courses are designed for self-paced learning. How does one debate if there is no one to debate with? We tackle that in the next question
Q: Can a forum be used in a self-paced course?
A: Yes, with a great deal of planning and assistance from technology. For example, if the technology can work like gaming rooms, a student can enter the forum and “wait” for another student (or group of students) to “enter” the room. This would require a course that is not completely linear, so students can go back to that module when there are enough students in the forum to make it work efficiently.
Q: Can a research paper be used to demonstrate competency?
A: Yes, but again that depends on how the assignment is constructed and the competency it is measuring. In general they are a learning activity–the student dives deeply into a particular subject and learns about it. But if there is a competency for demonstrating the ability to find and cite sources then a research paper then the that assignment would be good method for assessing the competency. However, generally speaking, the assignment should also include a presentation of the research with an extensive Q & A session (defend their conclusions)—to insure the student has mastered the material, and not merely regurgitated material.
Q: Should we assign points for homework?
A: Homework should never be used to measure an outcome/competency. Homework does not measure mastery. It is practice and therefore should not have points associated with it. Homework should, however, receive feedback or be followed by a practice quiz that provides feedback. (test for understanding)
Projects, Scenarios, Simulations:
Q: We use these for learning activities, but can they also be used for measuring mastery?
A: Yes, projects, scenarios, and simulations can provide for authentic (or close to authentic) summative assessments. With scenarios and simulations, there should be a small margin or error for demonstrating competency (or mastery). A scenario or simulation that was used for training purposes, should not be reused, as is, for assessment of mastery. They should be changed somewhat. Also, while there might’ve been hints or other feedback provided during the training, hints should not be allowed in the summative assessment. Projects should be constructed in such a way as to mimic one that a student would need to complete in the “real” world, with the same or similar expectations.
Interviews & Observations: live, video and audio:
Q: What other methods can be used for assessing competency?
A: Interviews are excellent methods for assessing mastery level learning, especially if the student is not given the questions beforehand and is not allowed to use notes. Observing the student, particularly in clinical/classroom/workplace environments are also excellent ways of measuring competency.
Q: Can observations also be a useful tool for helping students gain competency?
A: Yes, observations can be an excellent tool for learning. For example, when an athlete or performer watches a video of themselves, they can observe where they need improvement and/or what is working well for them. It is an excellent tool for self-assessment. For example, having students record themselves giving a speech, allows them to hear how many “ums” they say, as well as their cadence. When others use an observation to provide constructive feedback, it can also be an excellent learning tool.
Blogs and other forms of Journaling:
Q: What about blogs, reflections and personal journals–those are mostly for processing and learning, right?
A: Journals and blogs can be excellent tools for measuring certain types of competencies, especially when they are accompanied by an artifact of learning. For example, if a student in a language acquisition program creates an audio recording of themselves speaking in the language, and accompanies that with a reflection that includes: what they are saying, why they chose that, how it is expressed culturally, etc, you get a much better understanding of the student’s grasp of the language. Reflections and blog posts can also be powerful tools for assessing mastery gained in internships. For example, in a blog post the student can discuss their experiences and what they are learning. Tests can no be designed to measure somethings that only blog posts and personal reflections can reveal.
I hope you found this helpful. Feel free to post your suggestions in the comment section.
In the consultations I often provide for various online programs, I’ve seen a particular problem over and over again in regards to the integration of technology: educators begin integration from a focus on the technology. At conference after conference I hear educators talking about reaching students through new technologies, with once again the focus on the technology. I’d like to give some real life examples of how this can sometimes be short-sighted and problematic.There are two questions that should guide the integration of any technology: what problem is it meant to solve and/or what objective does it match to?
Plenty of instructors are adding mobile components to courses because “students want to use their cell phones” or because they read its an upcoming technology. What the educators do not understand is that this is about accessing their coursework using their mobile device, instead of a computer or laptop. It does not mean they want you to create an assignment that requires the use of a mobile device! If you are requiring students to have a mobile device for a particular course, then it better well have a measurable objective associated with that requirement. One example, would be the need for majors in Geographical studies to use GIS applications. This is related to program objectives for their career.
Here are two examples of a courses that required the use of mobile devices–one that was a good integration, and one that was a poor integration. School A offered a course on Ethical Uses of Technology for Educators. This course required students to have a mobile device. The objectives associated with this requirement were developed to insure that teachers became familiar with mobile devices and the unethical ways they could be used (advertently or inadvertently) in a classroom. The students in this course were given activities that required them to test how easy it would be to use a cell phone in unethical ways, and to reflect on how this would impact the classroom. This is a good integration of mobile technologies. The second example is a poor use: the course was on American Music. There were four objectives to the course, all of which required an understanding of different aspects of music. A mobile component was added because the developers wanted an innovative course, and that intent was to enable students to upload and download music on their handheld device. Not a single one of the objectives of the course had any reason to require knowledge of this skill, nor anything related to mobile technologies in general. The purpose in including this activity was so that the professors could research whether students would use a mobile device. This is an example of a very poor understanding of integration. To make it clear, a better way to do this would be to insure the course was hosted on a site that could be accessed and interacted with via a mobile device.
Support for third party applications can also become a problem relatively quickly, and once again I’m referring to unnecessary third party programs such as various web 2 programs. Instructors get angry that the helpdesk can’t or won’t provide support for whatever application they choose to use, but there are thousands, if not millions of them out there. At smaller colleges, where courses are taught exclusively by the faculty member that developed them, the instructor should insure that they are familiar with the application before requiring students to use it. At large colleges, where courses are developed by a team and taught by adjuncts, the problems are much bigger, and third party applications need to be selected more carefully. Adjuncts assigned to teach the course may not be familiar with the program, or they may have their own favorites, and may not be willing to learn it a third-party because its a program the developer likes. It is also unfair to expect the helpdesk to learn them all and be prepared to assist students. Again, third party apps should be chosen when they are needed to solve a particular problem, when the helpdesk is willing to support it and/or there is a particular course objective tied to the use of that application.
Access and course objectives should always be the first considerations. Activities and assessments need to be directly related to those objectives. Technologies should be chosen with those in mind. True student-centered teaching does not require particular technologies because they are cool, but because they will assist the student in achieving the course objectives and provide greater access, otherwise we may be putting undo demands on the students.
Here are some important questions that can help guide the integration of technology:
1. Is the addition of the required technology needed in order for students to achieve course objectives?
2. Will the technology decrease or impede access in anyway?
3. Is support available for students who have difficulty with the technology?
4. Will learning to use the technology detract from students learning the required content of the course?
We all need to make our courses more collaborative and more engaging for our students. We also need to have students exposed to the various technologies they will encounter in the work environment. We just need to insure that the technologies we choose help, not hinder, learning.
About a year ago I was working with an institution on its adoption of an Online Course Evaluation tool. It was a rather long process which included getting information on options and associated costs, piloting the program, responding to the concerns of the faculty about how the surveys would be used, setting up the tool so that the surveys would go out in emails to students, developing the survey tool itself (the questions) and then analyzing the results of the surveys to determine if the questions did, in fact, give us the type of information we were looking for.
Sadly, one of the greatest hurdles was the reluctance on the part of faculty to have a survey at all. I say “sadly” because such surveys can be invaluable for an instructor, a course designer, and the institution itself. If nothing else, well constructed surveys can help in determining if a program has redundancies, have difficulty navigating the course, or if students are unclear about the learning objectives.
Course surveys should occur, at the very least, at the end of a course. Even though information gathered at that time would be too late to make the course better for those students, the information can improve the course for the future students, and perhaps even improve the program as a whole. Too many instructors have the belief that students do not know enough to be able to determine whether there were clear objectives, whether they were met, whether there were unnecessary redundancies, or even if the work assigned was too much (or too little) for a 3 credit course. My experience has taught me that students do know, and we should ask them.
The ideal would be to survey students at the very beginning of a course, at the midterm, and at the end. The purpose of the beginning survey is to determine what students may already know or have experienced, and what they expect from the class. The midterm survey should be given to determine if the course is meeting its goals, and to determine if the students are struggling with a particular part of the course. The benefit of this is that necessary changes can be made before the course is done and it’s too late. We’ve already discussed the final evaluation.
If the questions are framed right, the survey can even cause students to reflect on their learning–to think about what they’ve done and learned. It can be used to develop metacognitive skills. There is a free tool available to assist with the development of these surveys. The site is called Student Assessment of Learning Gains, and you can access it here: http://www.salgsite.org/ You can use the wizard to create your own surveys, and you can browse a library of surveys to get an idea of what other instructors are asking.
Assessing what we are doing, all along the way, is an important part of insuring we are meeting our students needs and expectations.