Home » Administration

Category Archives: Administration

The Importance of Editors for Insuring Quality Courses

note reading "Let's eat Mom!"

What’s for dinner?

Few people would publish a book or paper without having it first reviewed by an editor, or even simply someone they trust to have a good handle on the language. Yet, most colleges and universities publish online courses without a once over by an editor.  I realize that some of this is due to an incorrect interpretation of academic freedom, but I’m not going to discuss that in this article. The consequence of errors in a public publication falls primarily on the author (or journal), however the consequence of errors in an online course primarily falls on the recipient (the student).

My experience, in the field of course design and development, is that while schools build teams of instructional designers, and invest in multimedia programs to enhance their online courses,  few have course editors on their teams. And, while I would never claim to be capable of being an editor myself, my experience as a student in online courses, and as a course reviewer,  is that many courses have significant errors in grammar, spelling, and syntax. In one of my graduate courses, the language was so bad it became a huge distraction.  While I had a genuine respect for this individual’s knowledge of the subject matter,  it was clear the individual had great difficulty in communicating clearly in writing. The large number of errors made some parts of the content almost unreadable, and the descriptions of various assignments extremely unclear. This problem existed throughout the entire semester. A course editor could’ve made all of this better for everyone concerned: the professor and the students.  This is by far not the only course I’ve seen with these issues, though this was probably the worst.

Recently, I experienced this same problem in a MOOC offered by Stanford. The course had probably spent months in development. It was nicely designed, with high-end and entertaining video. But, there were numerous blatant grammatical errors in the audio, and text.  A large portion of the forum posts complained of the language errors–some quite strongly. What made this particularly problematic, was that many of the students were not native English speakers.

What are the negative consequences of poorly written courses? Well, the first is that credibility can be negatively impacted. Cognitively, it increases the effort a student needs to process the information–sometimes significantly. In terms of Cognitive Load Theory, it increases extraneous load–which means it is more difficult for the student to learn the material–it negatively impacts learning. But, it can also give students the incorrect information, and it can create problems with assessment of student learning. For example, if a student misunderstands a poorly written question, or misunderstands what the assessment is asking the student to produce, then the student’s answer is an inaccurate measurement of what the student knows or is capable of doing. The assessment is invalid.

Often, we can use the context of a sentence to help us correctly interpret a poorly written sentence. For example, if we read a note: “Let’s eat Mom”, we would know the sentence is not suggesting we eat Mom, but rather “Let’s eat, Mom”. However, grammatically incorrect content, with which we have little familiarity or limited understanding, may not be  easily interpreted correctly.

It is important for institutions to think of online courses as published materials, and insure they are reviewed by editors to the same degree that the school would like all of their public facing materials reviewed.

 

PLA, Test-Outs, and CBE Pretests

Standard VIII, in CAEL’s Ten Standards for for Quality Assurancestates “Fees charged for assessment should be based on the services performed in the process and not determined by the amount of credit awarded.” Anyone who has gone through the portfolio process of earning credit through PLA appreciates the work involved in articulating the learning and being assessed on it. Some PLA programs include an option that allows students to test out of a course. Usually this involves the construction of a well thought out assessment, built by the faculty and assessed under very secure circumstances. The student generally does not pay the tuition for the course, but a set fee for the exam. There are standardized exams for this process (CLEP, ACE, etc), but colleges often construct their own as well. How, then, does this compare with pretests in competency based learning?

A lot of thought needs to be put into the pretest–what is it purpose? Is the student paying for credit hours, or is there a set fee regardless? Many colleges and universities, offering CBE, have gone with alternative tuition fees. Considering Standard VIII above, this makes sense. But what if the institution hasn’t changed the way they charge tuition? If the pretest occurs before the student enrolls in the course, and provides the student with the option of testing out of the course (is given credit for the course ), then there is no conflict, especially if the student is allowed to complete any final artifact of competency that enrolled students will complete. However, if the pretest occurs within a CBE course, and allows a student to  “test out”, then there is a conflict, because the student is paying by the credit hour. In this situation, my own recommendation is that students who can prove they’ve achieved the required level of competency in any given course, should be allowed to test out. Schools developing a CBE program, should consider having this option available for all students.

Pretests that occur within the context of a CBE course, should have the primary purpose of being a diagnostic tool. They can provide students (and instructors) with important information, especially because by its very nature CBL is self-paced. A well constructed pretest can and should have questions that are matched to the competencies in the given course. Additionally, the competencies themselves should have an allotment of time associated with them–the time it would take the average student to cover the content required to master the competency (what we once thought of in terms of carnegie units). When the student completes the pretest, he/she should receive a report that indicates their current level of competency in each of the related areas (so they know how much time and energy they may need to spend in a given competency), and the actual calculation of time they would need to devote to the tasks. This will allow the student, who may need more than a semester to complete the course, the opportunity to request more than a semester and/or know how many courses they can reasonably complete in a given amount of time.

In summary, test-outs and PLA should be used to give credit up front for a particular course/competency, while pretests should be considered diagnostic and not used for awarding credit. All CBE programs should offer some sort of PLA credit. This does not mean that a student, who has not tested out, can’t move through the course rapidly; they can still skip practices and content (if they choose) yet will need to complete the graded assessments associated with all the given competencies within the course.

I’ve attached a suggested template for planning your pretest, which you can download: Pretests_Maps

And here is a  small sample of the kinds of data that properly constructed pretests can provide to students: table showing question #, competency, difficulty, score, time on question

data showing how student scored against competencies and approximate time they will need to spend in order to master the competency

Let’s really talk about assessment

picture of highway ramp with white line on the left and yellow line on the rightThis is the picture of an off-ramp taken from the rear window, and cropped out of the picture is a car that has been totaled.  A car in which a group of teens were killed when they got on the highway going the wrong way. If you were to get on an offramp, going the wrong way, how would you know? The clue is the yellow line on the right side and white line on the left. The white line should always be on the right-hand side, and the yellow on the left. I’ve found that most people don’t know this, and yet that piece of knowledge can be a matter of life and death.

If this piece of information is critical, then shouldn’t it be taught in driver education? And, if it is taught in driver education should we make sure it is assessed? A common method of assessment is the use of quizzes constructed of random questions selected from a larger test bank. This type of assessment does not guarantee that the student will be assessed on critical information. If the drivers test consisted of a subset of random questions selected from a test bank, students may or may not get a question on white lines. Should they? And, if so, should it matter if they get it correct?

These are the kinds of things we should consider when constructing assessment. What is it that students need to have mastered in a course in order to pass? Do we directly assess it, and do we insure students are not passed along until they actually master it? If a course does not have specific competencies that a student needs to master, as some individuals will claim about their course, then why are we asking students to take it? There must be some skill or knowledge, at some level of mastery (even if it is on the beginner level) that we want students to get from that course.

Today, I came across a wonderful document on CBE posted on the Merrimack High School web site:

http://mvhs.mvsd.k12.nh.us/MathPresentation.pdf.

Why use competencies: 90%takeoff, 86%turbulence, 83% panel knowledge, 50% landing. Still passing, but would you want to get on the plane?

I took a screenshot of one of the slides, because I think it illustrates an important point about the weights we assign to various

assessments. I’ve seen a good many rubrics, and far too many of them don’t weigh the most important items properly. I’ve seen a good many category weights that allow students to completely fail the final exam and still pass the course–sometimes with a grade as high as an B.

A good part of the problem education now faces is due to the way we assess education, the way we credential it as well, and maybe even the way we think about it.  Much  more thought needs to go into what we assess, how we assess it, and how much weight we put on those assessments.

Constructivism & Competency Based: can they co-exist in the same classroom

Constructivism and constructivist pedagogies has been a topic of debate and research for decades, and many programs have been developed based on a certain definition of it’s meaning.  David Jonassen et all describe this in the following way: “The constructivist sense of ‘active’ learning is not listening and then mirroring the correct view of reality, but rather participating in and interacting with the surrounding environment in order to create a personal view of the world. Constructivists engage the learners so that the knowledge they construct is not inert, but rather usable in new and different situations.” (Jonassen, Davidson, Collins, Campbell & Haag, 1995)

Jacqueline Grennon Brooks defines it this way: Constructivism is basically a theory … It says that people construct their own understanding and knowledge of the world, through experiencing things and reflecting on those experiences.” (Grennon Brooks, 2004)

Problem-based learning, and scenario-based learning (a form of problem based learning) are methods used in constructivist classrooms–creating authentic learning experiences. Students in a constructivist classroom are expected to self-directed, and a focus is on developing metacognition in the student in order for them to become self-directed learners. However, when we discuss these types of learning, or types of instructions, we are not talking about how these are assessed. The premise is that everyone’s experience will be different, that truth is relative to the individual and that no two people will experience the same event in the same way. But, while difference individuals may tackle a problem differently, there often is only a small subset, if not a single, truth (or correct “answer”).

It makes sense when we think about this with regards to medicine–you want a physician that can apply their knowledge and skills to complex circumstance, to use critical thinking skills to solve problems, to be competent, and to come up with the right solution (to make the correct diagnosis)–but what about an editor, or an accountant, or an artist? Can we apply the same expectations to these fields and to learning in general.

Competency-based learning, like constructivism, requires students to be able to construct their own knowledge and to be able to apply that knowledge (and skills) to authentic, varying and new experiences. Competency-based education requires that students have a depth of knowledge and an acquisition of skills that can be applied not simply regurgitated. While constructivism focuses on the method for acquiring knowledge and skills, CBE focuses on how that is assessed and what triggers progression . They not only can coexist, but without students constructing their own knowledge, it should be, theoretically, impossible for them to pass a competency-based program.

Grennon Brooks, J. (2004). Workshop: constructivism as a paradigm for teaching & learning. Retrieved from http://www.thirteen.org/edonline/concept2class/constructivism/index.html

Jonassen, D., Davidson, M., Collins, M., Campbell, J., & Haag, B. B. (1995). Constructivism and computer-mediated communication in distance education.American Journal of Distance Education9(2), 7-26. doi: 10.1080/08923649509526885

Student Feedback for Assessment

About a year ago I was working with an institution on its adoption of an Online Course Evaluation tool. It was a rather long process which included getting information on options and associated costs, piloting the program, responding to the concerns of the faculty about how the surveys would be used, setting up the tool so that the surveys would go out in emails to students, developing the survey tool itself (the questions) and then analyzing the results of the surveys to determine if the questions did, in fact, give us the type of information we were looking for.

Sadly, one of the greatest hurdles was the reluctance on the part of faculty to have a survey at all. I say “sadly” because such surveys can be invaluable for an instructor, a course designer, and the institution itself. If nothing else, well constructed surveys can  help in determining if a program has redundancies, have difficulty navigating the course, or if students are unclear about the learning objectives.

Course surveys should occur, at the very least, at the end of a course.  Even though information gathered at that time would be too late to make the course better for those students, the information can improve the course for the future students, and perhaps even improve the program as a whole. Too many instructors have the belief that students do not know enough to be able to determine whether there were clear objectives, whether they were met, whether there were unnecessary redundancies, or even if the work assigned was too much (or too little) for a 3 credit course.  My experience has taught me that students do know, and we should ask them.

The ideal would be to survey students at the very beginning of a course, at the midterm, and at the end.  The purpose of the beginning survey is to determine what students may already know or have experienced, and what they expect from the class. The midterm survey should be given to determine if the course is meeting its goals, and to determine if the students are struggling with a particular part of the course. The benefit of this is that necessary changes can be made before the course is done and it’s too late. We’ve already discussed the final evaluation.

If the questions are framed right, the survey can even cause students to reflect on their learning–to think about what they’ve done and learned. It can be used to develop metacognitive skills. There is a free tool available to assist with the development of these surveys. The site is called Student Assessment of  Learning Gains, and you can access it here: http://www.salgsite.org/ You can use the wizard to create your own surveys, and you can browse a library of surveys to get an idea of what other instructors are asking.

Assessing what we are doing, all along the way, is an important part of insuring we are meeting our students needs and expectations.